Psychometric Properties of Cognitive Assessment in Amyotrophic Lateral Sclerosis: A Systematic Review

Introduction

Insight about the cognitive changes that might occur in amyotrophic lateral sclerosis (ALS) has increased during the last decades. Such changes are now recognised as one of the integral features that characterizes this multisystem disease.¹ Cognitive changes may influence ALS patient survival,² caregiver burden,³ and affect decision-making processes regarding treatment.⁴ These findings stress the importance of accurately identifying those affected and detecting the extent of cognitive change.^1,5 Early and repeated assessment may help healthcare professionals to better prepare families, enable individually tailored support, and provide timely opportunities for making essential decisions before impairment emerges.^1,4

Recent studies show that the prevalence of cognitive changes in ALS varies between 30% and 62%.^6,7 This wide range could be due to real differences in ALS patient characteristics (ie disease stage) or different tests used to assess cognitive changes.^8–10 During the past years, more focus has been directed on cognitive assessment and Strong et al have recently published revised diagnostic criteria for ALS-specific cognitive impairment.¹ To better define the extent and nature of changes, the authors differentiate between brief assessments and screening in clinical settings, assessment for prognostic stratifications in clinical trials and assessment focusing on research. A diagnosis of ALS-specific cognitive change depends on evidence of changes in executive function, including social cognition and/or language.¹ Although controversy, evidence of memory dysfunction in patients with ALS are increasing.^1,7 Changes in these cognitive functions may occur separately or in combinations with each other and should be assessed formally.¹ As a minimum, Strong et al suggest assessment of Phonemic/letter fluency early in the course of ALS, and that all patients undergo screening with the ALS Cognitive Behavioural Screen (ALS-CBS) or the Edinburgh Cognitive and Behavioural ALS Screen (ECAS).^1,5,11 The necessity of further assessment may depend on whether changes in cognition are registered and/or the setting in which an assessment takes place.¹ Moreover, the patient’s physical disability, risk of exhaustion and possible practice effect should be taken into account. Further, Strong et al provide a list of tests, sensitive to ALS and recommended for evaluation of executive function, language and memory. According to the authors, extended assessment should include neuropsychologists or speech and language pathologists to support administration and interpretation of cognitive profiles and may thus not be feasible.¹ Despite these efforts, not all instruments/tests are available everywhere yet and more effort and knowledge are needed among both clinicians and researches. Better application of the Strong criteria,¹ using tests covering a wide spectrum of cognitive symptoms may help to better understand ALS-specific cognitive changes.⁹

To aid healthcare professionals to make informed choices as they select tests capable of detecting ALS-specific cognitive changes, this review has the following objectives: (i) to list cognitive tests applied in ALS-specific healthcare (Phase I); (ii) to provide a descriptive synthesis of the psychometric properties of tests that were evaluated in a population of ALS patients (Phase II).

Materials and Methods

To ensure transparent reporting, we report our systematic review according to the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA).¹²

Protocol and Registration

The protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) before any study procedures were undertaken. The protocol was last updated on February 12, 2020, at the PROSPERO site, available from: http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42017055603.¹³

Eligibility, Exclusion, and Inclusion Criteria

Eligible for inclusion in this systematic review were research articles, published in full text and reporting on cognitive function in patients with ALS. Patients had to be diagnosed using recognised criteria common to ALS-specific healthcare.^14–16 We excluded articles about studies that employed experimental cognitive paradigms and paradigms used in combination with PET/CT/FMRI when focused on neuroanatomical questions of ALS. We also excluded articles reporting on children or animal. Included in Phase I were all relevant articles, regardless of the publication date, language they were written in and published. Of these results, we selected in Phase II, only those articles reporting on psychometric properties of cognitive tests that were evaluated in patients with ALS.

Information Sources

Systematic online searches were carried out in the Cochrane Library, McMaster⁺, Epistemonikos, Medline, Embase, SveMed+, CINAHL, AMED, OT seeker, PEDro and PsycINFO. We also checked the reference lists of systematic reviews we identified for additional relevant articles.

Search

A trained librarian carried out database searches. Our first search was carried out in February 2017 in order to identify the cognitive tests used in ALS-specific health care (Phase I). Search terms covered subject headings and free text words for the following three concepts: ALS/MND, cognition and specific cognitive domains known in ALS, and assessment tools. A second search was performed in December 2019 in order to find validating studies of cognitive tests used in patients with ALS (Phase II). In this second search we added a forth concept; psychometrics, to the searches in the main medical databases (Medline, Embase, PsycINFO, CINAHL, Cochrane Library), according to the search strategy filters developed by the Hedges Team for detecting clinical prediction studies.^17,18 See Supplementary material 1 for the search strings and the exact terms that we used for Medline and Embase in the first and the second search.

Study Selection and Data Collection Process

In Phase I, all references from the first search were screened. In Phase II in order to find validating studies, included references from Phase I and new references from the second search were screened. Two of the authors independently screened all references and extracted the data. Consensus for inclusion was reached by discussion among the authors. For articles written in languages others than English, we consulted professional colleagues who are knowledgeable about health terminology in those languages. For eight articles, we contacted the authors for additional clarifying information.

Data Items

In Phase I, a neuropsychologist listed all the tests with sufficient supporting references used in the selected studies and classified them according to the cognitive domains they address; attention, concept formation and reasoning, construction and motor performance, executive functioning, learning and memory, perception, social cognition, verbal functions and language skills. This procedure was mainly based on well-established work in the field of neuropsychology^19,20 as well as on test manuals or aim (s) of the studies of the included articles. In this evaluation, we were aware that the classification could to some extent vary, depending on the perspective of the investigators. Batteries, rating scales, inventories and screening tests were categorised separately when all subtests were applied. Since several articles, however, reported on just a few subtests from the complete test battery, and names used seemed not always identical with the original ones, names of the batteries were assigned to the above domain categories. When a complete battery was used, it was assigned separately.²⁰

In Phase II, we extracted various characteristics of the included articles and published information on each identified test regarding psychometric properties. We were particularly interested in studies that evaluated the feasibility, internal consistency, reliability (test–retest reliability, inter-rater reliability), content validity, construct validity (structural validity and hypotheses testing), criterion validity and responsiveness of tests, as most of these evaluation parameters are specifically addressed in the Consensus-based Standards for the Selection of Health Status Measurement Instruments (COSMIN).²¹ In Table 1, definitions and explanations of the above-mentioned psychometric properties (except feasibility) are presented together with recommendations of preferred statistical methods and minimum sample size to include in analyses.

Table 1 List of the Reported Psychometric Properties in Phase II – Definitions, Explanations and Recommendations of Preferred Statistical Method and Minimum Sample Size Included in Analyses

Assessment of Study Quality (Risk of Bias)

In Phase II, we evaluated the risk of bias and methodological quality of the studies in the included articles. We used the four steps of the COSMIN checklist.²¹ Only the standards corresponding to the psychometric property of interest were used and accompanied by specific explanations and instructions given in the COSMIN checklist.²¹ As feasibility is not part of the COSMIN checklist, we did not evaluate the methodological quality of studies that assessed the feasibility of the tests they used.

Level and Synthesis of Evidence

For tests identified in Phase II, we first calculated a quality score (excellent, good, fair, or poor) per measurement property, according to the scoring system developed by Terwee and colleagues.²² For each test of a study, psychometric properties were assessed and were rated as adequate (+), not adequate (-) or unclear (?). A rating of 0 was given when no information was available in the article. We then synthesised individual ratings using the criteria proposed by the Cochrane Back Review Group, leading to categories of strong evidence, moderate evidence, limited evidence, conflicting or unknown evidence.²³

Ethical

Ethical approval and consent to participate are not applicable for this systematic review and analysis since no research subjects were recruited. This study adheres to the principles of the Declaration of Helsinki.²⁴

Results

In Phase I for identifying cognitive tests, we included 319 articles. The flow of information through Phase I of the systematic review is presented in Figure 1. Of these 319 articles identified in our first search in 2017, 35 articles reported information on the psychometric properties of cognitive tests used in patients with ALS. In our second search conducted in 2019, we identified 11 new references reporting information on the psychometric properties of cognitive tests used in patients with ALS. A total of 46 articles were therefore included in Phase II of our systematic review. See Figure 2 for flow of information through Phase II of the systematic review.

Figure 1 Flow of information through Phase I of the systematic review: Identifying cognitive tests. Notes: Adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:10.  Creative Commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.12Abbreviation: ALS, amyotrophic lateral sclerosis.

Figure 2 Flow of information through Phase II of the systematic review: Evaluating psychometric properties. Notes: Adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:10.  Creative Commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.12Abbreviation: ALS, amyotrophic lateral sclerosis.

Phase I: Cognitive Tests Identified for Use in ALS-Specific Healthcare

Table 2 lists the most frequently reported; cognitive tests classified into eight cognitive domains, test batteries, and rating scales evaluated in the included articles. The total number of tests referenced within each cognitive domain and the total number of times a test is referenced are shown.

Table 2 List of the Most Frequently Reported Cognitive Tests, Test Batteries and Scales Used in Patients with Amyotrophic Lateral Sclerosis (ALS) in 319 Articles Identified in Phase I

A complete list of all cognitive tests identified in Phase I are presented in Supplementary material 2. We observed that for some tests, several different versions were used, which indicates that the tests were in the process of being developed or had been revised (eg the Wechsler Adult Intelligence Scale), or that the tests were modified or adapted specifically for use in patients with ALS.

Phase II: Cognitive Tests That Were Evaluated in Patients with ALS

Table 3 summarises the characteristics of articles and studies reported therein that were included in Phase II of this review. All of the included studies reporting on feasibility of tests, focused on the suitability of specific tests for use in patients with motor and speech impairment. The authors of those studies indicated that the Edinburgh Cognitive and Behavioural ALS screen (ECAS) might be more suitable for patients with ALS than the Frontal Assessment Battery (FAB) and the Montreal Cognitive Assessment (MoCA) screening test. Table 4 lists cognitive tests evaluated in patients with ALS identified in the first (2017) and second search (2019) of the review, and examined in Phase II according to their cognitive domain classification. Available in Table 4 is also information in which country each test has been validated.

Table 3 Characteristics of Articles Included in Phase II

Table 4 List of Cognitive Tests Evaluated in an ALS Population, Identified in the First and Second Search (2017 and 2019) of the Review and Examined in Phase II, Classified into Cognitive Domains

Phase II: Assessment of Study Quality

Assessment of the methodological quality of included studies using the COSMIN checklist²¹ was for each psychometric property of interest (see Table 1 for definitions and explanations):

Internal Consistency

None of the nine studies evaluating internal consistency explicitly described whether the tests used were based on a reflective or formative model. Nonetheless, all of the nine studies calculated Cronbach’s alpha or Item response theory (IRT), methods that are both adequate parameters of internal consistency. Four studies assessed the unidimensionality of the test they used. Of these four studies, two conducted the unidimensional subtest separately, and all four had a sample size of greater than 50.

Reliability

None of the five studies that assessed reliability evaluated test–retest reliability. For evaluation of inter-rater reliability, the raters were independent in four studies. Three studies used intraclass correlation coefficient, a preferred statistical measure of reliability. Two studies used Pearson’s correlation, which does not take systematic error into account. Two studies had a sample size of less than 50 participants.

Content Validity

One study assessed the content validity of the test they used; however, they did not provide a detailed description of how this was done.

Construct Validity – Structural Validity

None of the four studies evaluating structural validity explicitly described whether the tests they used were based on a reflective or formative model. Three studies used IRT methods and followed the recommendation of checking the dimensionality of the items. One study that relied on classical test theory did not use factor analysis for this check, which is the preferred statistical method to determine the structure of the instrument.

Construct Validity – Hypothesis Testing

Of 28 studies evaluating hypothesis testing, 26 studies formulated a priori hypotheses. Specification of expected direction (positive or negative) of correlations one expected, or quantification of the range of expected differences between groups or scores were lacking completely or described only vaguely in 23 studies. Of the 14 studies that used hypothesis testing to investigate the convergent validity of tests, all 14 studies provided references for the comparator test; however, eight of these studies lacked information about measurement properties. In 27 studies, correlations were used. In seven studies, the sample size was inadequate.

Criterion Validity

In 24 of 29 studies evaluating criterion validity, we found that the authors compared the scores of the new test to a widely used one, not the original longer version. Statistic methods like correlation, receiver operating characteristics and sensitivity/specificity were used in all 24 studies. The sample size were inadequate in eight studies.

Responsiveness

All the three studies evaluating responsiveness tested the subjects at least two times during the assessment period and reported the time interval between measurements. Precise descriptions of what occurs during the follow-up were not adequately described in any of the included studies. To evaluate responsiveness, all three included studies tested a specific hypothesis. All three studies formulated hypotheses a priori, but the hypotheses were formulated only vaguely. Two studies reported p-values for the hypothesis testing they did, even if this should be avoided since it is not relevant to examine whether correlations statistically differ from zero.⁷⁰ The sample size in the analyses were too small in all the included studies.

Phase II – Synthesis of Evidence

The results from the synthesis of the evidence on the quality of the test psychometrics in Phase II of this review are presented in Table 5.

Table 5 Levels of Evidence: Quality of Psychometric Properties and Feasibility of Tests in Articles Identified During Phase II

We found that the highest level of evidence was supported for the screening tests; Addenbrooke’s Cognitive Evaluation (ACE) and Frontal Assessment Battery (FAB), and the Reading the Mind in the Eye Test (RME), a test of social cognition. We found support for a moderate level of evidence for the ECAS and the MoCA screening tests and the Awareness of Social Inference Test, another test of social cognition. It should be noted that when evaluating the quality of screening tools, only the ECAS has alternative versions limiting practice effects. Moreover, it is available in several languages. Of all the included articles in Phase II, we observed that the greatest number of psychometric properties of interest was evaluated for the ACE and the ECAS. The Mini-Mental State Examination (MMSE) showed inconsistent levels of evidence.

Discussion

In this review, we had two aims: (i) to compile a comprehensive list of all cognitive tests used in ALS-specific healthcare and (ii) to provide a descriptive synthesis of the psychometric properties of tests validated in patients with ALS.

Our literature search and analyses indicated that a large number of tests are being used in ALS-specific healthcare. Of these tests, however, only a fraction have been evaluated for use within a population of ALS patients. This may lead to different conclusions about a patient`s cognitive functioning.²² Although commonly used, some tests, especially the MMSE, have produced conflicting evidence when it comes to assessing ALS-specific cognitive changes.^25,34,56,67

To date, of the tests used to evaluate cognitive changes in ALS, none fulfils all of the essential measurement properties stipulated by the COSMIN checklist.²¹ We did, however, observe acceptable measurement properties for four screening tests; the ACE, FAB, ECAS and MoCA. Feasibility studies, however, showed that the ECAS might be more suitable, and hence preferable, for screening cognitive function in ALS patients compared to the FAB and MoCA, a finding that is in line with that of another systematic review.⁷¹ While several screening tests have been developed to detect cognitive changes, the ECAS has been designed specifically to assess cognitive changes in patients with ALS.⁵ Strong et al recommend that either the ECAS or the ALS-CBS be administered to all patients.¹ However, in our review, we found the psychometric properties of the ALS-CBS poorly evaluated. Our analysis revealed that, of all the screening tests evaluated in the included studies, the psychometric properties of the ECAS and ACE were the most scrutinised. It should be noticed, however, that effort has been made to provide alternate versions of the ECAS.⁷² The challenge in recurrent testing is that the disease may sometimes deteriorate quickly. Therefore, the tests should be easy to perform, but not be easily remembered and learned at the same time. This enables recurrent testing which is desirable to follow the development of cognitive changes over time. This way, significant practice effects can be reduced if not eliminated. Moreover, translations in several languages would allow international multicentre studies, which could increase knowledge and patient care.

Of two tests of social cognition evaluated in patients with ALS, we found that the highest level of evidence was supported for the RME. Although not specifically developed for use in patients with ALS, the RME accounts for potential physical disability as it can be conducted in either written or spoken.⁷³ The RME is recommended by Strong et al for evaluation of the ability to read the emotions of others, a social cognitive process most likely in patients with ALS who meet the criteria for frontotemporal dementia.¹

Our data suggest that the ECAS may be preferred over other screening tests for validly and reliably screening cognitive change in patients with ALS and that the RME may be suitable for the extended assessment of social cognition. The advantage of ECAS is that it is an ALS-specific test and therefore may show changes that are characteristic for ALS. However, our data did not allow us to give clear recommendations on which specific tests to use for extended assessment of other cognitive changes that may occur in patients with ALS. Similar methodological flaws recurring across the included studies preclude us from making such recommendations, and also prompt us to highlight the need for researchers and clinicians to pay particular attention to the methods applied in psychometric studies. Hence, there is a great potential for improving the methodological quality of studies that aim to evaluate the measurement properties of a cognitive test used in ALS-specific healthcare.

Study Strengths and Limitations

The strengths of the current study lay in the rigorous quality assessment achieved by the use of the COSMIN checklist, a widely used and validated tool to assess the risk of bias in evaluating measurement properties of a test.²¹ Also, in combination with the COSMIN checklist, we used recommended quality criteria²¹ to synthesise the level of evidence.²² Other strengths are that a trained librarian conducted the literature search and that the searches were relatively comprehensive and exhaustive. However, we did not consult all databases, nor grey literature, a strategy that future reviews might consider.

This review did not take studies in healthy controls into account. A more accurate reflection of the psychometric properties of each test would include studies with healthy controls. Therefore, the reliability and validity of some tests may be underestimated. The analyses demonstrate that in particular, screening instruments have been evaluated so far, and more research is needed with regard to evaluate further cognitive/neuropsychological tests. Another limitation can be the inclusion of articles since 1985, since this may risk confusing the results as different versions of cognitive tests may have varying psychometric properties. We argue that our aim was to provide a comprehensive overview possibly also including results from countries where not all tests/versions are available yet. Furthermore, the analyses in the present study are limited to the assessment of cognition as one aspect of behavior. We acknowledge the importance to investigate tests assessing emotional aspects of behavior as a future.

The quality of included studies was rated as fair to poor in many cases, which limits the conclusions we can draw. Available data did not allow us to conduct meta-analyse and we could not perform assessments of heterogeneity. Thus, at this point, we can only give guarded advice for tests that should be used in ALS-specific healthcare. We are aware that the presence of publication bias may have affected our results. Finally, some evaluator bias may have also affected our result, because three of the authors of this review are involved in the validation of the Norwegian version of the ECAS.⁷⁴

Conclusions

To the best of our knowledge, this is the first study to systematically identify and evaluate tests that are used to measure the cognitive change in ALS and those tests that have been validated in patients with ALS. The screening test, ECAS and the social cognition test, RME; may have some advantages over other tests that have been used for assessing cognitive change in ALS patients. Psychometric properties of tests that were evaluated in a population of ALS patients have mainly been examined in screening instruments. Thus, more research regarding the validation of comprehensive neuropsychological assessment seems appropriate to implement the recent ALS criteria.¹