Reliability Evidence for the Gibson Assessment of Cognitive Skills (GACS): A Brief Tool for Screening Cognitive Skills Across the Lifespan

Introduction

Since the close of the 19th century, cognitive testing has evolved and advanced through the pioneering work of Spearman, Thurstone, Cattell, Horn, Carroll, Kaufman, and more.^1,2 As intelligence theories have become more diverse in recent years, many professionals agree that traditional aptitude assessments sustain cultural bias and miss important cognitive measures.^3–5 Technological advances have further confounded the paradigms of intelligence testing as digital and computerized assessment methods lack human interaction and require computer experience and proficiency.^6,7 Recent literature reveals the need for improved cognitive assessment to incorporate human delivery, comprehensive cognitive measures, and lifespan norming.^8,9

In keeping with the most widely accepted view of cognition, the Cattel-Horn-Carrol theory,¹⁰ it is important to consider a comprehensive analysis of cognitive abilities in intelligence testing, including specific auditory processing skills.^11–13 Recent research reveals evidence of need for ongoing development of cognitive assessment to more accurately incorporate more aspects of the CHC model.^14,15 We argue that development of such tests should consider a lifespan approach, the ability to screen briefly, and a simplified scoring and reporting system. Indeed, there is a dearth of such ability.

Further, the absence of comprehensive assessment of auditory processing skills in cognition batteries is pervasive. Auditory processing underlies the ability to read, write, participate meaningfully in conversations, acquire a foreign language, and navigate performance bottlenecks such as driving a car.¹⁶ Therefore, identifying deficits in auditory processing skills should be a critical aspect of cognition assessment.

Cognitive performance has historically been measured through formal assessments administered by highly trained clinicians and interpreted by doctoral level practitioners. There are several gold-standard assessment tools in widespread use today. Cognitive tests designed for children include the Cognitive Abilities Test (CogAT),¹⁷ Differential Ability Scales (DAS-II),¹⁸ Naglieri Nonverbal Ability Test (NNAT3),¹⁹ Otis Lennon School Abilities Test (OLSAT),²⁰ Stanford Binet 5,²¹ Wechsler Intelligence Scale for Children (WISC-5),²² and Woodcock Johnson (WJ-IV).²³

For adults, there are also many available cognitive assessments such as the Kaufman Adult Intelligence Test (KAIT),²⁴ the Wechsler Adult Intelligence Scale (WAIS-IV),²⁵ and the Woodcock Johnson (WJ-IV).²³ However, there is a distinct lack of widely available cognitive testing tools for assessment across the lifespan within one battery. The Kaufman Brief Intelligence Test (KBIT-2)²⁶ is applicable for ages four through 90 yet lacks important cognitive skill measures such as auditory processing. The Reynolds Intellectual Assessment Scales (RIAS-2)²⁷ is normed for ages three to 94 years yet does not contain comprehensive measures within one assessment. For example, the critical skill subtests for memory and processing speed are add-ons. Offshoots of traditional cognitive assessment have emerged that measure the more nuanced executive function skills such as the NEPSY-II²⁸ and the Cognitive Assessment System (CAS2).²⁹ Finally, there are several tests that are specifically designed to screen for cognitive decline in the aging, such as the Montreal Cognitive Assessment MoCA³⁰ and the Mini Mental State Exam (MMSE).³¹ However, there are not many robust tools available to quickly screen performance in core cognitive skills in healthy populations or administered by non-clinical practitioners such as teachers. Further, the ability to use the same tool for screening across the lifespan is indeed limited. This limitation creates the need for practitioners to invest in multiple test batteries, frequently at a substantial financial cost to themselves or their organizations. It is indeed advantageous to have a screening tool that enables practitioners to assess cognition affordably without having to purchase multiple batteries to cover different ages. A lifespan tool can also enable researchers to use the same assessment in cross-sectional and longitudinal studies across age groups.

The Gibson Assessment of Cognitive Skills (GACS)³² is a paper-based cognitive screening tool based on CHC theory. The assessment—formerly called the Gibson Cognitive Test Battery—includes measures of working memory, processing speed, visual processing, logic and reasoning, and three auditory processing constructs: sound blending, sound segmenting, sound deletion, as well as word attack skills. The assessment includes lifespan norming, with paper-and-pencil human delivery, in one brief 30-minute test, thus filling several of the identified gaps in cognitive assessment including the need for a brief, lifespan cognition assessment tool that adequately measures multiple skills including auditory processing. The GACS is designed to be administered and scored by both clinicians as well as teachers, nurses, allied professionals, and researchers with basic training in administration of assessments, enabling rapid turnaround of results.

Each GACS subtest was developed following an extensive review of the literature and two decades of clinical practice and applied research on cognitive skills that underlie the ability to think and learn. The test itself was developed in alignment with the Cattell–Horn–Carroll theory of cognition with each subtest designed to measure one of the CHC broad constructs: fluid reasoning (Gf), working memory capacity (Gwm), processing speed (Gs), visual processing (Gv), auditory processing (Ga), and reading and writing (Grw). Subject matter experts in psychology, neuroscience, and education were consulted during development to validate the content of each subtest to provide assurance and agreement on including each test item, and to validate the alignment with CHC constructs. The GACS belongs to the same family of tests that includes the computer-based cognitive test called the Gibson Test of Cognitive Skills (GTCS). About 50% of the items overlap between the two tests and the statistical analysis methods conducted for the GTCS were replicated for the current study.³³

Along with the content and construct validity, the convergent validity of the individual subtests on the GACS has also been previously examined^34,35 indicating strong correlations with the WISC IV (Processing Speed, r = 0.40, Auditory Processing, r = 0.52, Visual Processing, r = 0.60, Working Memory, r = 0.73, Logic and Reasoning, r = 0.72); with the Woodcock Johnson III (Word Attack, r = 0.78, Processing Speed, r = 0.76, Visual Processing, r = 0.73); with the Detroit Tests of Learning Aptitude – 4 (Working Memory, r = 0.61, Logic and Reasoning, r = 0.69); and with the Lindamood Auditory Conceptualization Test (Auditory Processing, r = 0.63). The test was normed on a sample of 3490 children and adults between 2015 and 2019, and complete demographics are available in the technical manual.³⁶ Preliminary unpublished evidence on the internal consistency reliability and split-half reliability of the GACS had been previously collected but a comprehensive examination of multiple sources of reliability had not been conducted. Therefore, the sole aim of the current study was to examine and report newly collected sources of reliability evidence—including test–retest reliability, internal consistency reliability, and split-half reliability—for the GACS.

Materials and Methods

The current study of the Gibson Assessment of Cognitive Skills was guided by the Standards for Educational and Psychological Testing.³⁷ Ethics review and approval to conduct the study was granted by the Institutional Review Board (IRB) of the Gibson Institute of Cognitive Research prior to commencing data collection. All procedures were performed in accordance with the 1964 Helsinki Declaration and its later amendments. We conducted a series of investigations to examine evidence of test reliability including analyses of the internal consistency reliability evaluated through item analysis and the resulting coefficient alpha, the test–retest reliability estimated with Pearson correlation of two test administrations, and split-half reliability determined through the correlation of two halves of each subtest.

Measures

The Gibson Assessment of Cognitive Skills (GACS) is a paper-and-pencil battery of eight subtests that measure six broad cognitive constructs. A description of each subtest follows:

Processing Speed Test

The processing speed (Gs) test broadly measures fluency in task performance. Specifically, the task measures perceptual speed which is the speed in which similarities in visual stimuli can be searched and identified. The examinee is given a page of 38 rows of 6 sets of letter or number combinations of increasing difficulty and must find the matching sets in each row (see Figure 1). There are 38 possible points (one point per match) in this 2-minute timed test.

Figure 1 Example processing speed test items. Note: Permission to share this image has been granted by the test publisher LearningRx.

Working Memory Test

The Working Memory subtest measures both auditory and visual working memory capacity, or the ability to encode and keep visual and verbal information in one’s primary memory. An example of a visual stimulus is shown in Figure 2. The examinee hears an oral story and is then shown an unrelated picture. He studies each picture before the picture is removed by the examiner, and then answers questions about the story and the picture or previous stories and pictures. The test is a form of an n back task, where the examinee is asked questions about a picture or a story presented two to six items previously. For example, the examinee is told a brief story about a girl and her pet. Then, the examinee is shown the unrelated picture illustrated in Figure 2. Next, the examinee is asked a question about the story and then a question about the picture. He is told another brief story and shown another unrelated picture. Then, he is asked a question about a previous story and picture before being asked about the current story and picture. The stories and pictures become progressively more detailed. There are 24 total questions about 5 stories and 5 pictures for a total of 24 possible points.

Figure 2 Example working memory test item. Note: Permission to share this image has been granted by the test publisher LearningRx.

Visual Processing Test

The Visual Processing Test measures the ability to perceive visual patterns and to visualize how they look when rotated. For each item, the examinee is shown a completed puzzle and must identify the location of the individual pieces in the puzzle when separated and rotated (See Figure 3). There are 35 items that make up seven puzzles of increasing difficulty, for a total of 35 possible points.

Figure 3 Example visual processing test item. Note: Permission to share this image has been granted by the test publisher LearningRx.

Logic and Reasoning Test

The Logic and Reasoning Test measures inductive reasoning skill, or the ability to infer a rule based on the observation of a phenomenon. For each item, the examinee is shown a matrix of designs with a missing space and he must select from a set of designs the one that fits in the missing space. (See Figure 4). There are 17 matrices of increasing difficulty in this subtest for a total of 17 possible points. This subtest has a ceiling of 4 consecutive incorrect answers.

Figure 4 Example logic and reasoning test item. Note: Permission to share this image has been granted by the test publisher LearningRx.

Auditory Processing Test

The Auditory Processing Test measures the ability to analyze and manipulate sounds and sound patterns. This test includes three sections (Blending, Deletion, and Segmenting), each designed to measure the CHC auditory processing (Ga) constructs of phonetic coding (the ability to hear, blend, and segment words or sounds), memory for sound patterns (the ability to retain auditory codes), and speech sound discrimination (the ability to tell the differences in speech sounds without distortion). All three sections are delivered orally with items of increasing difficulty. The Sound Blending task requires the examinee to listen to the individual sounds in a nonsense word and then blend the sounds orally into a completed word. For example, the examiner says, “/a/-/p/” and the examinee must say, “ap.” The Sound Segmenting task requires the examinee to listen to a nonsense word and then say all the sounds in the word individually. For example, the examiner says, “gos’” and the examinee must say, “/g/-/o/-/s/.” The Sound Deletion tasks require the examinee to delete a given sound from a word and say the new word. For example, the examiner says, “Say Cat without the/c/sound” and the examinee must respond with “at.” There are 28 items on the Auditory Processing Test and a total of 90 possible points.

Word Attack Test

The Word Attack Test measures the broad CHC construct of basic reading-writing ability (Grw) and specifically grapheme to phoneme translation through phonetic and reading decoding. The examinee is shown lists of nonsense words (pseudowords) in increasing difficulty such as “baf” and “thrate” and is asked to read them aloud. There are 24 items on this subtest for a total of 24 possible points. The ceiling on this subtest is four consecutive inaccurate answers.

Sample

The sample (n = 103) for the current study consisted of children (n = 73) and adults (n = 30) between the ages of 6 and 80 (M = 20.2, SD = 17.7), 47.6% of which were female and 52.4% of which were male. The ethnicities included Caucasian (84.5%), Black (6.8%), Asian/Pacific Islander (3.9%), Hispanic (1.9%), and Native American (2.9%). Half the sample (n = 50) was recruited from existing clients of four clinical practices in Georgia, California, New Jersey, and Colorado. The other half of the sample (n = 53) was recruited through a medical practice in Colorado. Table 1 shows detailed demographic data for the sample. The minimum sample size required for the current reliability study was determined using the formula and tables introduced by Bonett³⁸ with the following parameters: the number of items is greater than 100, the null hypothesis set to coefficient alpha of 0.50, the alternative hypothesis coefficient alpha set to 0.70. The minimum sample size required to be able to reach 90% power is n = 84. Therefore, our sample size of n = 103 exceeds the minimum required.

Table 1 Demographics of Reliability Sample

Procedures

After obtaining ethics approval from the Gibson Institute of Cognitive Research Institutional Review Board (Protocol #20,200,207) prior to study commencement, recruitment took place in two phases. First, de-identified test data were collected from the four practices around the country. The test was administered to their clients as part of their intake assessment process. Then, recruitment of the additional participants began in Colorado through email flyers with an invitation to participate. Participants were screened and included based on the following criteria: age between 6 and 80, no gross impairment in hearing and vision, no previous diagnosis of dementia or intellectual disability, and ability to use a pencil without assistance. After obtaining written informed consent and child assent when appropriate, a master’s level clinician supervised by a doctoral level psychologist administered the test two times scheduled one week apart to 53 participants. Following collections of the test forms, the data were transferred to spreadsheets for analysis to calculate internal consistency reliability and split-half reliability on the whole sample (n = 103), and test–retest reliability for each subtest on the second half of the recruited sample (n = 53).

Data Analysis

We conducted our data analyses using SPSS Version 26 and JMetrik Version 4.1.1 software programs. After running descriptive statistics in SPSS to determine mean scores and demographic distributions, we began our analysis of internal consistency reliability in JMetrik. This procedure produced a coefficient alpha for each subtest. Next, we split the test items into odd and even halves and ran Pearson’s correlations in JMetrik for each subtest to calculate split-half reliability. Then, we used SPSS to run Pearson’s correlations between the two test administrations spaced one week apart to determine test–retest reliability.

Results

Internal Consistency Reliability

Coefficient alpha represents the internal consistency reliability of each subtest, or how well the test items correlate with each other. In addition to an overall coefficient alpha for each subtest, coefficient alphas are reported for two age intervals—children and adults. (Table 2). Overall coefficient alphas range from 0.80 to 0.94, producing a strong source of reliability evidence. The standard error of measurement (SEM) is also reported for each coefficient alpha, ranging from 1.5 to 2.4 overall.

Table 2 Internal Consistency Reliability Metrics for the GACS

Split-Half Reliability

Split-half reliability is the estimation of test reliability determined by correlating the scores on two halves of the test. To calculate the split-half reliability estimate, the sum of the even numbered items was correlated with the sum of the odd numbered items. A Spearman-Brown formula was applied to the Pearson correlation for each subtest to predict the overall reliability of the test. Because the standard split-half reliability calculation is not appropriate for a speeded test, the following alternative calculation was used for Processing Speed: r₁₁ = 1 – (SEM²/SD²). Overall, the split-half reliability coefficients ranged from 0.83 to 0.96, producing a strong source of reliability evidence. Split-half reliability coefficients for both children and adults are also reported. (Table 3)

Table 3 Split-Half Reliability Coefficients

Test–Retest Reliability

Test–retest reliability is an indicator of the temporal stability of the test across administrations. Pearson’s correlation of two sets of scores captured one week apart from the same participants are illustrated in Table 4 through 6 along with the mean and standard deviation of the test scores and retest scores, the mean score difference, and the difference in standard deviation units. Overall, the test–retest reliability coefficients ranged from 0.83 to 0.98 (Table 4), indicating strong test–retest reliability and stability across administrations. All correlations were statistically significant at p < 0.001.

Table 4 Overall Test–Retest Correlations

For adults ages 18 to 80, reliability coefficients ranged from 0.73 to 0.99, indicating strong test–retest reliability and stability across administrations (Table 5). All correlations were statistically significant at p < 0.001.

Table 5 Test–Retest Correlations for Adults

For children ages 6 through 17, reliability coefficients ranged from 0.89 to 0.97, indicating strong test–retest reliability and stability across administrations (Table 6). All correlations were statistically significant at p < 0.001.

Table 6 Test–Retest Correlations for Children

Discussion

The purpose of the current study was to examine the reliability of the Gibson Assessment of Cognitive Skills (GACS), a brief paper-based screening tool for assessing cognitive performance in ages 5 through adult. Although several sources of validity evidence had previously been collected for the GACS,^34,35 evidence of reliability of the test had not been comprehensively examined or published. The evidence collected in the current study indeed supports the reliability of the tool for screening cognitive performance across the lifespan.

First, the internal consistency reliability of the GACS is strong, with coefficient alphas ranging from 0.80 to 0.94 overall. The reliability coefficient should be at least 0.70, with 0.80 serving as the most suitable minimum threshold. In contrast, coefficient alphas higher than 0.95 suggest the presence of redundant items. The internal consistency reliability of the GACS, therefore, comfortably falls within the desirable range. Indeed, these coefficients are also stronger than several other published cognitive assessment measures. For example, several subtests on the WISC-V and DAS-II have coefficient alphas in the 0.70s.^18,22

Next, the split-half reliability coefficients of the Gibson Assessment of Cognitive Skills indicate strong internal consistency reliability across items as well, ranging from 0.83 to 0.96 overall. While the split-half reliability of two subtests fell below the desired threshold of 0.80 for the adults, the split-half reliability coefficients ranged from 0.85 to 0.96 for children. The adult sample was considerably smaller than the child sample, suggesting this result may be more of a function of the adult sample rather than a function of the test itself. In future research, it will be interesting to note if this same pattern is apparent.

A key finding in the current study was the stability of the GACS over time. The test–retest reliability correlation coefficients were strong and significant, ranging from 0.83 to 0.98 overall. The strongest correlation was across administrations of the auditory processing subtest (0.98) while the lowest correlation was across administrations of the working memory subtest (0.83). These correlations were significant and strong for both the child and adult samples and again exceed the test–retest reliability coefficients of other published cognitive assessments. For example, the test–retest reliability ranges from 0.75 to 0.94 for the WISC-V³⁹ and from 0.21 to 0.91 for the NEPSY-II.²⁸ Further, the test–retest reliability of the GACS also exceeds several popular cognitive screening tools for adults, including the CANTAB (0.17 to 0.86)⁴⁰ and CNS Vital Signs (0.31 to 0.86).⁴¹

Along with the strong sources of reliability evidence collected in the current study, the GACS is also the only published single cognitive screening assessment that measures three narrow abilities of auditory processing (blending, segmenting, deletion) as well as Word Attack skills. As the critical skills that underlie the ability to read and write, auditory processing as a key component of CHC theory is indeed critical to comprehensively assess. We were pleased with the strong reliability of this portion of the GACS in particular since it is a unique aspect of the test compared to other assessments in widespread use.

There are two limitations to the current study that are important to highlight. First, the adult sample was small and lacked a robust number of older adults. Although we found strong reliability evidence for adults in our analysis, it is possible that a larger sample would produce different results. Therefore, it is important that we conduct additional reliability research specifically with older adults. Another limitation to the current study is the absence of an examination of inter-rater reliability. Because the assessment is designed to be administered by both clinicians and non-clinicians, it is important to evaluate the reliability in scoring between experts and non-experts. This should be the focus of an additional research study on this assessment tool.

To date, evidence of reliability and validity has not been collected on the use of the GACS with clinical populations. Future research should include examination of discriminant and predictive validity with various clinical groups including brain injury and age-related cognitive decline. However, the evidence collected thus far suggests that the GACS is indeed a reliable brief screening tool for assessing cognitive skill performance across the lifespan.