Computerized screening for cognitive impairment in patients with COPD

Introduction

Previous studies found that a substantial number of patients with COPD have cognitive impairment.^1,2 The impairments are found in different cognitive domains, including memory, attention, and speed of information processing.^1–7 Impairment in cognitive functioning has been associated with negative outcome in COPD patients, including decreased health status⁷ and more health care use.^7,8 It has been suggested that cognitive impairment has a negative effect on outcome by interfering with treatment adherence.⁹ Disturbed medication adherence has been related to verbal memory impairment.⁵ Emery et al¹⁰ found that dropouts of pulmonary rehabilitation (PR) performed worse on cognitive flexibility than those who finished the program.

Although COPD patients with cognitive impairment appear to be at higher risk of negative outcome and might benefit from additional support, hardly any attention goes to identifying cognitive impairment in clinical care. This inattention may be partly due to the lack of time- and cost-efficient cognitive screening instrument.¹¹ Conventional cognitive assessments take, on average, 4 hours to complete and may be too long to implement in COPD care. Other short and easy-to-administer cognitive screening instruments such as the Mini Mental State Examination (MMSE)¹² and the Montreal Cognitive Assessment (MoCA)¹³ were used in studies with COPD patients.^3,7,14 However, the MMSE has been questioned as a cognitive screening instrument in COPD patients.^3,7,14 The MoCA seems an attractive instrument because it is easy and quick to administer. It, however, only gives a global impression of cognitive functioning. Other brief cognitive screening instruments that provide a more detailed impression of multiple cognitive domains are available.

In a previously published letter,¹¹ we referred to a computerized cognitive test battery that we used to assess cognitive functioning in COPD patients: Central Nervous System Vital Signs (CNSVS).¹⁵ This is an example of a brief way to screen for (severity of) cognitive impairment in multiple cognitive domains.¹⁶ The standardized test administration and the automatic scoring of the tests make computerized cognitive screening time- and cost-efficient, and highly standardized compared with paper and pen cognitive tests.

In this study, we look at the viability of CNSVS as a cognitive screening instrument in COPD patients. Such a cost- and time-efficient screening instrument is highly needed in clinical COPD practice.

Material and methods

Setting and selection criteria

Patients were recruited at Revant, PR center ‘Schoondonck’, Breda, the Netherlands (referred to as “Schoondonck”). Schoondonck is a tertiary care center that offers PR to patients with respiratory diseases.

A prior study in COPD patients referred to PR at Schoondonck¹⁷ showed us that the majority of the patients had at least 3 of the following characteristics: 1) severe to very severe airway obstruction global initiative for chronic obstructive lung disease stage of III–IV based on pulmonary function parameters; 2) limited exercise performance with a maximal power output (W_max) on a cycle ergometer of <50% of predicted; 3) severely impaired quality of life, defined as a St George’s Respiratory Questionnaire (SGRQ) total score of >50, and 4) a relatively high annual, self-reported exacerbation rate (>3 in the preceding year) and healthcare utilization (>2 pulmonary disease-related hospitalizations in the preceding year).

Inclusion criteria in this study were a COPD diagnosis and referral to the intensive, comprehensive PR program at Schoondonck.¹⁷ Patients were excluded from study participation in case of unfinished primary school.

Design

Cognitive performance of COPD patients was compared with CNSVS age-corrected norms based on healthy American people (n=1,600+). Dutch CNSVS norms were not available at the time of this study. In addition, the relation between cognitive performance and computer familiarity was examined. The data presented in this article are part of the baseline neuropsychological assessment of a randomized controlled trial to study the effects of PR on cognitive functioning, mood, anxiety, and quality of life of patients with COPD (ClinicalTrials.gov: NCT01682447). The study was approved by the medical ethical review board of the Saint Elisabeth Hospital, Tilburg, the Netherlands (file number NL33713.008.10). All participants provided written informed consent for this study.

Procedures

In Figure 1, procedures of this study are presented. Neuropsychological assessment was scheduled (based on randomization) during the pre-PR assessment week or during the first week of PR. Time to complete CNSVS took ~30 to 40 minutes. In our study, a test technician was present during the cognitive assessment to assist the patient. Measures of health-related quality of life, exercise performance, and pulmonary and clinical variables (“Measurements”) were collected in the pre-PR assessment week for all patients as part of normal clinical procedures. The time interval between those measures and the cognitive tests for those patients who had their neuropsychological assessment in the first week of PR varied from 1 to 15 weeks (median =4.6 weeks). The study was designed this way because this enabled us to design a randomized controlled trial with a waiting list control group very similar to the experimental control group.

Figure 1 Patient flow with reasons for exclusion and dropout. Note: CNSVS performance of patients measured in the pre-PR week and week 1 of PR were analyzed as one group (n=220). Abbreviations: CNSVS, central nervous system vital signs; COPD, chronic obstructive pulmonary disease; PR, pulmonary rehabilitation; wk, week.

Measurements

Cognitive functioning

Cognitive functioning was measured with a computerized cognitive screening called CNSVS.¹⁵ A Dutch translation of the Core-7-CNSVS screening battery was used, including 7 cognitive tests: verbal memory test, visual memory test, finger tapping test, symbol digit coding test, Stroop test, shifting attention test, and continuous performance test. Based on these tests, 7 cognitive domain scores were calculated (Table 1): memory, psychomotor speed, processing speed, reaction time, complex attention, cognitive flexibility, and executive functioning, and also the neurocognition index (NCI), a global cognitive measure. The report of a patient’s test results is automatically generated by the program, including raw (ie, actual) test scores, standard scores corrected for age, and percentile ranks. The standard scores are based on a normative database of healthy American people (n=1,600+). The mean standard score for each cognitive domain is 100 (SD ±15) and higher standard scores indicate better performance.^15,16 Standard domain scores >110 are “Above Average” and scores of 90 until 110 are “Average” and these scores indicate, respectively, high and normal cognitive functioning. Standard domain scores from 80 until 89 are “Low Average” and indicate slight deficit and impairment. Scores <80 are “Low” and indicate that moderate deficit and impairment are possible. Scores <70 are “Very Low” and impairment and deficit are likely.

Table 1 CNSVS cognitive tests and corresponding cognitive domains Notes: aThe mean of these domain scores is the NCI; bIf applicable. Abbreviations: CNSVS, central nervous system vital signs; CPT, continuous performance test; FTT, finger tapping test; NCI, neurocognition index; SAT, shifting attention test; SDCT, symbol digit coding test; VeMT, verbal memory test; ViMT, visual memory test.

Other variables

For an overview of the variables described here, Table 2. Sex, age, education, frequency of computer use, having a partner or not, and employment status were obtained via personal interview. Years of education was the sum of the years of formal education starting with primary school. Highest education level was classified with the 7-point rating scale of Verhage¹⁸ and summarized in 3 categories: low (ie, lower vocational education and below, Verhage 2, 3), middle (ie, intermediate vocational education, Verhage 4, 5), and high (ie, college and university, Verhage 6, 7). Unfinished primary school (ie, Verhage 1) was an exclusion criterion. To estimate the frequency of computer use, patients chose whether they never, sometimes, or frequently used a computer. Premorbid verbal intelligence was estimated with the Dutch adult reading test.¹⁹

Table 2 Demographic and clinical characteristics of patients Notes: Unless stated otherwise n=220; aminimum–maximum; bincluding CVA, lacunar infarct, subarachnoid hematoma, TIA, subdural hematoma. Abbreviations: 6MWDT, six-minute walking distance test; BMI, body mass index; CVA, cerebrovascular accident; DART, Dutch Adult Reading Test; FEV1, forced expiratory volume in 1 second; GOLD, Global Initiative for Chronic Obstructive Lung Disease; kPa, kilopascal; PaO2, partial oxygen pressure in the blood; PR, pulmonary rehabilitation; SGRQ, St George’s Respiratory Questionnaire; TIA, transient ischemic attack.

Forced expiratory volume in 1 second (FEV₁) was measured according to the guidelines of the American Thoracic Society (ATS) and the European Respiratory Society. FEV₁ percentage predicted was calculated with the reference equation from the European Community of Coal and Steel.²⁰ The arterial or capillary partial oxygen pressure was measured prior the PR, without additional oxygen supply. The number of exacerbations and hospitalizations in the year preceding PR was based on patients’ self-report. Comorbidities were based on patients’ self-report and their medical records.

Health-related quality of life was measured with the total score of the SGRQ.²¹ Scores ranged from 0 (no impairment) to 100 (maximal impairment). Lower scores indicated better health status.

Exercise performance was measured with the maximal power output (W_max) obtained in an incremental test on a cycle ergometer (Ergoselect 1000, Ergoline, Bitz, Germany). The 6 minute walking distance test followed the ATS guidelines²² in a 40 meter corridor. Three tests were performed on alternate days. The results of the highest test are reported here.

Statistical analyses

Age-corrected standardized CNSVS domain scores were used in all analyses. One-tailed 1-sample z-tests for each domain score were performed to test if cognitive performance of the COPD patients was worse than the normative mean (M =100, SD ±15). To control for multiple testing, a Bonferroni correction was applied with alpha set at (0.05/8=) 0.00625. Cohen’s d effect sizes²³ were calculated with a pooled SD weighted for the sample size of the group of COPD patients and the normative database.

Besides the mean group performance, the percentage of patients with scores in the impaired range (ie, standard score ≤79) was calculated for each CNSVS domain. Additionally, the number of impaired CNSVS domains per patient was calculated. Executive functioning was excluded from this calculation, because of the large overlap with cognitive flexibility (ie, the executive functioning domain score is based on the same shifting attention measures that are also included in the calculation of the domain of cognitive flexibility, Table 1). For the patients who did not complete all CNSVS tests, the sum of the impaired domains was based on the available cognitive domain scores.

Analysis of variance was employed to compare cognitive performance between patients who never, sometimes, and frequently used computers. Due to the large differences between group sizes, Hochberg’s GT was applied as a post hoc test.²⁴ This analysis was performed with SPSS 22.0 (IBM Corp., Armonk, NY, USA) and alpha was set at 0.05. Effect size omega-squared was calculated.^24,25

Results

Patient flow and characteristics

Patients were recruited from January 2011 until July 2013. From the patients referred for PR during this period (n=610), 290 patients did not fulfill the inclusion and exclusion criteria at some point (Figure 1 for more details). Of the eligible patients (n=320), 100 patients (31.3%) declined to participate or dropped out of the study (Figure 1). CNSVS cognitive assessment was scheduled either during the pre-PR assessment week (n=111) or during the first week of PR (n=109). The 2 groups were analyzed together as 1 group (n=220), because assessments were prior to or at the very beginning of PR for both groups and patients were randomly allocated to either one of the 2 groups. Characteristics of the patients that participated in CNSVS are presented in Table 2.

Completion rates per cognitive domain varied from 212 (complex attention) to 220 patients (processing speed) and CNSVS tests were fully completed by 205 patients (93.2%, 105 females, 100 males). Reasons for incomplete results were the inability to perform tests (n=5, eg, arthrosis interfering with psychomotor speed tests), extreme fatigue (n=2), misunderstanding of test instructions (n=4), or unwillingness to complete a test (n=3). One patient’s score on complex attention was excluded. The score was >6 SD below the mean and was therefore considered unreliable.

Cognitive functioning

Age-corrected mean CNSVS domain scores of patients were all significantly lower than the normative mean (Table 3). Processing speed differed the most from the normative mean with a medium effect size and mean performance that fell in the Low Average category. On the other 6 domains, the mean group performance fell in the Average category. Medium effect sizes were found on memory, complex attention, and cognitive flexibility. Small effect sizes were found for psychomotor speed, executive functioning, reaction time, and the global cognitive measure NCI.

Table 3 Cognitive domain scores of COPD patients compared to the normative mean with z-tests and percentage impaired per cognitive domain Notes: aCohen’s d effect size, where 0.2≤ d <0.5 is small, 0.5≤ d <0.8 is medium, and d ≥0.8 is large23; bOne-sample z-test, M =100, SD =15; cp<0.00625, one-tailed; dStandard score ≤79 is impaired. Abbreviation: NCI, neurocognition index.

With regard to impairment, most patients were impaired on processing speed (28.6%), while reaction time was least impaired (10.7%) (Table 3). Figure 2 presents the percentages of patients in the different score categories for each cognitive domain. Overall, 3 or more CNSVS cognitive domains were impaired in 19.5% (n=43) of the patients, 2 domains were impaired in 15.5% (n=34), and 1 domain was impaired in 16.8% (n=37), while 48.2% (n=106) of the patients performed in the normal range on all cognitive domains.

Figure 2 Percentages of patients per score category per CNSVS domain. Abbreviations: CNSVS, central nervous system vital signs; NCI, neurocognition index.

Computer familiarity and cognitive functioning

A significant effect of self-reported frequency of computer use was found in all CNSVS domains and NCI (p<0.05), except for complex attention (p>0.05, Table 4). Although an overall effect of frequency of computer use was found on memory, post hoc analyses showed no significant differences between the 3 groups. The group that never used a computer performed significantly worse than the group that frequently used a computer on psychomotor speed (p=0.001), cognitive flexibility (p=0.030), executive functioning (p=0.038), and NCI (p=0.005), while no other significant post hoc differences were found on these domains. On reaction time, the group that never used a computer performed significantly worse than the groups that sometimes (p=0.033) and frequently (p=0.001) used a computer, while the sometimes and frequent groups did not significantly differ. Post hoc analyses for processing speed showed significant differences between the 3 groups, where the group that frequently used a computer performed the best, while patients who never used the computer performed the worst (never vs sometimes, p=0.015; sometimes vs frequent, p=0.001; never vs frequent, p<0.001). The effect size of computer use was large for processing speed, medium for psychomotor speed, and small for reaction time, cognitive flexibility, executive functioning, memory, and NCI. The percentage of patients with 1 or more impaired cognitive domains was 44.9% in the group that frequently used a computer, 48.3% in the group that sometimes used a computer, and 76.2% in the group that never used a computer.

Table 4 Comparison of cognitive domain scores between never, sometimes, and frequent computer users with ANOVA Notes: Data in df are presented as: df between groups, df within groups. ap<0.05; bOmega squared effect size where 0.010≤ ω2 <0.059 is small, 0.059≤ ω2 <0.138 is medium, and ω2 ≥0.138 is large.25 Abbreviations: ANOVA, analysis of variance; NCI, neurocognition index.

Discussion

Although widely-used, to our knowledge, this is the first study using CNSVS in COPD patients. This study aimed to examine the viability of this brief computerized cognitive test battery as a screening instrument for cognitive impairment in COPD patients by looking at the completion rate, the effects of computer familiarity, and by comparing CNSVS results with the results of prior research using paper and pen cognitive tests.

This study showed that 205 (93.2%) of the patients completed all tests included in this cost- and time-efficient computerized cognitive test battery. Slightly more than one-half (51.8%) of the COPD patients referred for tertiary PR were impaired on at least 1 cognitive domain measured with CNSVS. Processing speed was the domain on which the highest percentage of patients (28.6%) scored in the impaired range. COPD patients performed significantly lower than expected from healthy people on all cognitive domains. Yet, the mean domain scores fell within the (Low) Average category, masking the substantial number of patients performing in the impaired range (51.8%). This shows us that it is also important to look at results of individual patients instead of focusing on group results only when studying cognitive functioning in COPD patients. The frequency of computer use had an effect on all cognitive domain scores, except for complex attention. Overall, cognitive performance was worse for patients who reported a lower frequency of computer use, especially among patients who never used a computer. The effect was strongest on cognitive domains that included tests requiring more keyboard skills (processing speed and psychomotor speed), but was also found on cognitive domains that hardly required any keyboard skills (memory).

This amount of cognitive impairment is in accordance with a study by Cleutjens et al.³ The percentage of cognitive impairment (51.8%) we found was slightly lower than the 56.7% Cleutjens et al³ found among COPD patients in another center for tertiary PR in the Netherlands using paper and pen tests. Comparing the cognitive domains included in both studies, the prevalence of psychomotor speed was similar (17.0% vs 17.8%), while cognitive flexibility and memory were lower in our study (23.5% vs 43.3%, 20.6% vs 33.3%). Perhaps our more conservative cut-off of −1.4 SD for cognitive impairment instead of the −1.0 SD applied by Cleutjens et al³ resulted in finding the lower prevalence of cognitive impairment. Also, CNSVS memory tests were recognition tests, which are usually easier than the active recall memory tests used in the other study.

The high completion rate of CNSVS and the similarity of our data with those of standard paper and pen tests support that this short, computerized battery is viable for screening cognitive deficits in COPD patients.

Minimal influence of computer familiarity was expected because only simple computer skills (ie, use of the space bar, shift and number keys) are necessary to execute the CNSVS tests, most tests start with a practice trial, and a test technician was present during the cognitive assessment to assist patients in case of difficulties. Nevertheless, our findings are in line with results of a previous study²⁶ that participants with less computer experience performed worse on CNSVS tests than more experienced sex-, age-, education-, and race-matched participants, especially on tests requiring more computer skills. These findings suggest that unfamiliarity with computers might lead to worse performance on a test, possibly due to slower reaction time, and underestimation of someone’s true cognitive functioning. However, unfamiliarity with computers was also related to worse performance on paper and pen tests,^27,28 and less cognitive decline was found among computer users compared with non-users of computers.²⁹ Therefore, another explanation could be that people who hardly or never use a computer might indeed have truly worse cognitive functioning than people who use a computer more frequently. This issue cannot be solved by the current study and needs more attention. For now, CNSVS results of patients with little experience with computers should be interpreted carefully, especially with regard to the domains processing speed, reaction time, and psychomotor speed that require more computer skills.

This study has some limitations. CNSVS memory tests include memory recognition tests, while no active recall is included. Active recall is different from, and in general, more difficult than memory recognition. Impaired active recall might be overlooked if only memory recognition tests are included and may have important clinical implications for COPD patients. For example, the application of treatment strategies might be easily disturbed in case of impaired active recall. A memory test, including active recall could be added to CNSVS.

In addition, CNSVS software automatically corrects cognitive domain scores for age, but not for education. Because education and cognitive test results are found to be positively related,³⁰ not correcting for education might have resulted in an underestimation of cognitive functioning in patients with lower education, while it might have resulted in an overestimation in patients with higher education. An American healthy control database was used to compare the results of Dutch COPD patients because no Dutch normative CNSVS data were available. Future research should study CNSVS results in a sample of healthy Dutch participants to further validate the use of CNSVS in the Netherlands and the effects of education level on CNSVS performance.

With regard to generalizability, this study provides data of cognitive functioning of a large sample of patients with moderate to very severe COPD referred for specialized PR. Pulmonary physicians may not refer patients with overt cognitive dysfunctions to Schoondonck, because of the intensity of the PR program. This might have resulted in an underestimation of cognitive impairment in COPD in general. When participants were accepted for PR at Schoondonck, no additional exclusion criteria with regard to comorbidities (eg, cardiovascular diseases) and exacerbations were applied. Comorbidities are highly prevalent among COPD patients referred for tertiary PR³ and a high frequency of exacerbations is an important reason to participate in PR at Schoondonck. Excluding patients for these reasons was expected to seriously limit the generalizability of our study results. Due to the choice of including patients with additional comorbidities, the results of this study provide insight in the cognitive impairments of COPD patients referred for tertiary PR, instead of cognitive impairments due to COPD per se.

Conclusion

Since a substantial part of COPD patients have cognitive impairments, and because cognitive impairment is related to negative outcome and treatment adherence in this population, identification of this impairment will be an important additional way to improve managing this disease and enhance quality of life in patients. This study showed that a brief, computerized cognitive screening battery might be a time- and cost-efficient way to identify COPD patients with cognitive impairment. Such a screening instrument is highly needed in clinical COPD practice. Results of patients with little computer skills should be interpreted carefully.

Acknowledgments

The authors thank all the patients for their effort to participate in this study. In addition, we thank the Schoondonck staff for their effort in recruiting patients for this study and scheduling the neuropsychological assessments. We thank Dr Karin Gehring for comments on an earlier version of the manuscript and Dr Wilco Emons for his statistical advice, and the research assistants for their help with the neuropsychological assessments.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.		Dodd JW, Getov SV, Jones PW. Cognitive function in COPD. Eur Respir J. 2010;35(4):913–922.
2.		Schou L, Ostergaard B, Rasmussen LS, Rydahl-Hansen S, Phanareth K. Cognitive dysfunction in patients with chronic obstructive pulmonary disease – a systematic review. Respir Med. 2012;106(8):1071–1081.
3.		Cleutjens FA, Franssen FM, Spruit MA, et al. Domain-specific cognitive impairment in patients with COPD and control subjects. Int J Chron Obstruct Pulmon Dis. 2017;12:1–11.
4.		Grant I, Heaton RK, McSweeny AJ, Adams KM, Timms RM. Neuropsychologic findings in hypoxemic chronic obstructive pulmonary disease. Arch Intern Med. 1982;142(8):1470–1476.
5.		Incalzi RA, Gemma A, Marra C, Capparella O, Fuso L, Carbonin P. Verbal memory impairment in COPD: its mechanisms and clinical relevance. Chest. 1997;112(6):1506–1513.
6.		Incalzi RA, Gemma A, Marra C, Muzzolon R, Capparella O, Carbonin P. Chronic obstructive pulmonary disease. An original model of cognitive decline. Am Rev Respir Dis. 1993;148(2):418–424.
7.		Dodd JW, Charlton RA, van den Broek MD, Jones PW. Cognitive dysfunction in patients hospitalized with acute exacerbation of COPD. Chest. 2013;144(1):119–127.
8.		Chang SS, Chen S, McAvay GJ, Tinetti ME. Effect of coexisting chronic obstructive pulmonary disease and cognitive impairment on health outcomes in older adults. J Am Geriatr Soc. 2012;60(10):1839–1846.
9.		George J, Kong DC, Stewart K. Adherence to disease management programs in patients with COPD. Int J Chron Obstruct Pulmon Dis. 2007;2(3):253–262.
10.		Emery CF, Schein RL, Hauck ER, Schein RL, MacIntyre NR. Psychological and cognitive outcomes of a randomized trial of exercise among patients with chronic obstructive pulmonary disease. Health Psychol. 1998;17(3):232–240.
11.		Campman CA, Sitskoorn MM. Better care for patients with COPD and cognitive impairment. Lancet Respir Med. 2013;1(7):504–506.
12.		Folstein MF, Folstein SE, McHugh PR. ‘‘Mini-mental state.’’ A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12(3):189–198.
13.		Nasreddine ZS, Phillips NA, Bédirian V, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005;53(4):695–699.
14.		Villeneuve S, Pepin V, Rahayel S, et al. Mild cognitive impairment in moderate to severe COPD: a preliminary study. Chest. 2012;142(6):1516–1523.
15.		CNS Vital Signs, LLC. CNS vital signs interpretation guide. CNS vital signs Website. Available from: http://www.cnsvs.com/WhitePapers/CNSVS-BriefInterpretationGuide.pdf. Accessed February 24, 2017.
16.		Gualtieri CT, Johnson LG. Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Arch Clin Neuropsychol. 2006;21(7):623–643.
17.		van Ranst D, Otten H, Meijer JW, van’t Hul AJ. Outcome of pulmonary rehabilitation in COPD patients with severely impaired health status. Int J Chron Obstruct Pulmon Dis. 2011;6:647–657.
18.		Verhage F. Intelligentie en leeftijd: onderzoek bij Nederlanders van twaalf tot zevenenzeventig jaar [Intelligence and age: research study in Dutch individuals aged twelve to seventy-seven]. In: Fokkema SD, Kouwer BJ, van de Loo KJM, Mulder M, Snijders JTh & Vuyk R, editors. Bijdragen tot de Psychologie. Assen: Van Gorcum; 1964;4. Dutch.
19.		Schmand B, Bakker D, Saan R, Louman J. De Nederlandse Leestest voor Volwassenen: een maat voor het premorbide intelligentieniveau. [The Dutch reading test for adults: a measure of premorbid intelligence level]. Tijdschr Gerontol Geriatr. 1991;22(1):15–19. Dutch.
20.		Quanjer PH, Tammeling GJ, Cotes JE, Pedersen OF, Peslin R, Yernault JC. Lung volumes and forced ventilatory flows. Report working party standardization of lung function tests, European Community for Steel and Coal. Official statement of the European Respiratory Society. Eur Respir J Suppl. 1993;16:5–40.
21.		Jones PW, Quirk FH, Baveystock CM, Littlejohns P. A self-complete measure of health status for chronic airflow limitation. The St. George’s Respiratory Questionnaire. Am Rev Respir Dis. 1992;145(6):1321–1327.
22.		ATS Committee on Proficiency Standards for Clinical Pulmonary Function Laboratories. ATS statement: guidelines for the six-minute walk test. Am J Respir Crit Care Med. 2002;166(1):111–117.
23.		Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.
24.		Field A. Discovering Statistics Using SPSS. 3th ed. London: SAGE Publications Ltd; 2009.
25.		Kirk RE. Practical significance: a concept whose time has come. Educ Psychol Meas. 1996;56(5):746–759.
26.		Iverson GL, Brooks BL, Ashton VL, Johnson LG, Gualtieri CT. Does familiarity with computers affect computerized neuropsychological test performance? J Clin Exp Neuropsychol. 2009;31(5):594–604.
27.		Fazeli PL, Ross LA, Vance DE, Ball K. The relationship between computer experience and computerized cognitive test performance among older adults. J Gerontol B Psychol Sci Soc Sci. 2013;68(3):337–346.
28.		Tun PA, Lachman ME. The association between computer use and cognition across adulthood: use it so you won’t lose it? Psychol Aging. 2010;25(3):560–568.
29.		Xavier AJ, d’Orsi E, de Oliveira CM, et al. English longitudinal study of aging: can internet/E-mail use reduce cognitive decline? J Gerontol A Biol Sci Med Sci. 2014;69(9):1117–1121.
30.		Lezak MD, Howieson DB, Bigler ED, Tranel D. Neuropsychological Assessment. 5th ed. New York: Oxford University Press; 2012.