Clinimetric properties of the electronic Pain Assessment Tool (ePAT) for aged-care residents with moderate to severe dementia

Introduction

Accurate and objective pain assessment is a foundation stone of optimal pain management. This is particularly relevant for patients with impaired cognition, such as those with dementia, whose ability to report or verbalize pain is compromised.^1,2 In this population group, there is currently a large unmet need in relation to pain assessment, resulting in undetected and untreated pain, which then contributes to worsening cognition (eg, delirium) and behavior (eg, agitation and aggression), inappropriate prescribing (eg, antipsychotics), and adverse drug effects (eg, death).^1,2 Furthermore, suboptimal pain assessment and management compromise the quality of life of patients with dementia and constitute a significant burden for their families and carers.^1–3 In order to improve the current situation, the electronic Pain Assessment Tool [ePAT; also known as PainChek] was developed, which is a point-of-care smart device-enabled application that uses automated facial (video) analysis in conjunction with clinical indicators to evaluate the presence and intensity of pain in those with communication difficulties, including dementia.⁴ This application was designed to improve the objectivity and accuracy of assessing pain in patients with dementia, ultimately leading to effective pain management for this disadvantaged group.

In July 2017, the ePAT application was approved as a Class I medical device by the Therapeutic Goods Administration in Australia and received the CE mark in Europe “. . . to assess and monitor pain in people who cannot verbalise such as people with dementia or communication difficulties.”⁵ To date, the application has been validated in a total of 74 residents aged 60–98 years across five residential aged-care facilities in Western Australia.^4,6 In this setting, it has shown strong psychometric properties, including concurrent validity, discriminant validity, and interrater reliability.^4,6

Clinimetric properties, such as predictive validity and clinical utility (CU), are also important criteria in evaluating new pain-assessment tools, because they determine the usefulness and applicability of the tool in the clinical setting.⁷ Literature data indicate that a large number of psychometric evaluation studies of pain-assessment tools have neglected this aspect.^2,8,9 Zwakhalen et al point out that further research should test validity, reliability, and CU of existing pain-assessment tools with the view of improving them.⁸ Furthermore, a more recent systematic review by Lichtner et al concluded that no single pain-assessment tool can be currently recommended in patients with dementia on the basis of lack of evidence concerning validity and CU to assess pain in patients with cognitive impairment.²

The CU of a pain-assessment tool is an essential clinimetric parameter that goes beyond the analytical, technical, or even diagnostic accuracy performance of the tool.¹⁰ Testing CU in fact provides more insight into potential health benefits and outcomes,¹⁰ especially in comparison to existing options, in this case the Abbey Pain Scale (APS), which is currently a widely used (ie, silver) standard for assessing pain in patients with dementia in Australia and other countries.^11,12 Furthermore, CU provides important information in relation to how useful the tool is in assisting the decision-making process about the patients, eg, administering an analgesic drug when a patient is in pain.¹³

In addition to our aim of examining CU, the focus of this paper is also to assess the predictive validity of the ePAT. This is relevant, given the significance of data in evidencing the potential ability of the ePAT in predicting criterion-reference measures of the APS, which are a key reason for assessing this important clinimetric property.^13–15 Specifically, this study aimed to analyze the predictive validity of the ePAT with a view toward discriminating and comparing between pain on movement and at rest with reference to the APS.

Methods

Ethics

This research was approved by the Human Research Ethics Committee of Curtin University, Western Australia (HREC: HR10/2014) and Mercy Health, Victoria (R15/50AC) in Australia. The study was also registered with the Therapeutic Goods Administration under the Clinical Trial Notification scheme (CT-2016-CTN-04886-1 v1). Proxy (third party) written informed consent was provided by legal representative(s) of participants involved, due to the latter’s impaired cognitive capacity to provide their own consent. No additional data were collected for the current study, as all data were sourced from a previously published study.⁶ Therefore, there was no burden on staff or residents. Data were deidentified to ensure confidentiality.

Study design, setting, and inclusion criteria

Data were extracted from a previous validation (observational) study of the ePAT, which was conducted in two residential aged-care facilities in Perth, Western Australia over 10 weeks between January and April 2017.⁶ Pain assessments were done twice (at rest and postmovement) in pairs by ePAT rater and APS raters (nursing staff of the residential aged-care facilities). Raters performed assessments independently, and were blinded in terms of one another’s assessment and pain therapies received by the residents. All pain assessments were performed inside the facilities between 1-4pm while the residents were receiving their standard clinical care. This was either done during sitting, or recumbent positions to reflect rest (non-nociceptive) conditions, or during walking or transfer as examples of movement (nociceptive) conditions. Residents’ cognitive level was assessed prior to the study using Psychogeriatric Assessment Scales – cognitive (PAS-Cog), which has been validated in people with dementia.¹⁶ PAS-Cog scores categorize the degree of cognitive impairment according to increments of 0–3 = minimal, 4–9 = mild, 10–15 = moderate, and 16–21 = severe. Cognitive data were extracted from the electronic medical records of the residents.⁶

Residents were selected if they met the inclusion criteria: ≥65 years of age, living in facility for ≥3 months, PAS-Cog score ≥10, had a diagnosis of dementia by a geriatrician, and a medical history or presenting complaint(s) linked to painful condition(s), such as arthritis. Residents were excluded if they were medically unwell or unable to display intact facial expressions. Further details of the methodology employed in this study, including inclusion and exclusion criteria of residents and raters, have also been discussed in a previous publication.⁶

Statistical methods

Descriptive statistics (eg, mean, median) were reported for pain scores, while inferential statistics (eg, CI) were used to determine the level of significance. The receiver-operating characteristic (ROC) curve was used to determine cutoff scores for the presence of pain, which were classified in a binary format (yes/no). The ROC curve is a graphical representation of the sensitivity versus 1 – specificity for a dichotomous variable.^17,18 The curve is a method of estimating the best balance between sensitivity and specificity using multiple measurements, and enables the selection of an optimal threshold value (cutoff point). Besides cutoff points, ROC curve analysis can be used to convert continuous or ordinal variables into dichotomous outcomes.^17,18 To derive the maximum value for these cutoff points, Youden’s index (sensitivity + specificity – 1) was also used.¹⁷ In our study, ePAT was the tool under investigation against the standard instrument, ie, the APS. Cross-tab calculations were made to enable univariate percentage agreement between ePAT and APS of the presence or absence of pain.

Predictive validity was measured using accuracy statistics (sensitivity, specificity, positive predictive value, negative predictive value, and positive and negative likelihood ratios, respectively). These values were calculated twice: before and after prevalence adjustment. Positive (+) and negative (-) CU index (CUI) values were calculated using Mitchell’s formula.¹⁹ In this study, we used the STARD (Standards for Reporting of Diagnostic Accuracy Studies) guidelines to report the clinimetric findings.²⁰ Calculations were based on comparison with the APS, which was used as a criterion reference in our study. Presence of pain was defined as per published scores of the APS (≥3) or ePAT (≥7), with absence of pain defined as APS score of 0–2 or ePAT score of 0–6.^4,11

All analyses were done using Microsoft Excel 2013 for Windows 7 Enterprise (Microsoft Corporation, Redmond, WA, USA), Clinical Utility Index Calculator,¹⁹ and MedCalc statistical software (version 17.4 for Windows; MedCalc Software, Ostend, Belgium).

Results

Participants

A total of 34 residents aged 68.0–93.2 years were included in our sample, with a mean of 85.5±6.3 years.⁶ The sample comprised 20 females (58.8%) and 14 males (41.2%), of whom 35.3% had Alzheimer’s dementia and 44.1% an unspecified type of dementia. With the exception of one resident, all (n=33, 97.1%) were Caucasian. A total of 27 residents (79.4%) had a diagnosis of severe dementia based on PAS-Cog scores (mean 19.7±2.5, range 11–21). The vast majority of the sample were either nonambulant (n=14, 41.2%) or ambulant with assistance (n=19, 55.9%). A full description of the sample data was published in a previous study.⁶

Pain data

A total of 400 paired pain assessments for 34 residents were included in the analysis.⁶ Of those, 303 episodes were classified as pain by the ePAT based on a cutoff score of 7 (Table 1). Values ≤6 were defined as no pain. On the other hand, 307 were deemed pain by APS based on a cutoff score of 3, whereby scores ≤2 were considered no pain (Table 2). Mean ePAT pain scores were significantly greater with movement (11.44±3.54, P<0.0001) than rest (8.33±3.34). During assessments, types of physical activities undertaken by residents were varied, and ranged from sitting to walking.⁶

Table 1 No pain vs pain using the electronic Pain Assessment Tool

Table 2 No pain vs pain using the Abbey Pain Scale

Clinimetric data

Predictive validity

Predictive validity was calculated using sensitivity, specificity, and accuracy before and after prevalence adjustment. In the context of this validation study (n=400 paired assessments), sensitivity represents the true positive rate of detecting pain by the ePAT when pain actually exists, as indicated by the APS. Specificity denotes the true-negative rate of detecting pain (ie, absence of pain) by the ePAT when no pain is identified according to the APS. For the ePAT, the presence of pain was defined as a score of ≥7 on the final scale.⁴ According to published scores of the APS, a total score ≥3 refers to presence of pain, while a score ≤2 refers to the absence of pain.¹¹

Accuracy statistics before prevalence adjustment

Diagnostic sensitivity and specificity were both calculated based on the results shown in Tables 1 and 2. Findings from accuracy statistics are presented in Tables 3 and 4. The ePAT has a pain-negative result if pain is absent, ie, in the case of no pain. Compared to the APS results in Table 2, 85 assessments were identified as true negative (TN) and 12 as false negative (FN). The ePAT has a positive pain result if pain exists, ie, the presence of pain. When compared to the APS, ePAT assessments were found to be 295 true positives (TPs) and eight false positives (FPs).

Table 3 Contingency (2×2) table for accuracy statistics Abbreviations: ePAT, electronic Pain Assessment Tool; APS, Abbey Pain Scale; TP, true positive; FP, false positive; FN, false negative; TN, true negative.

Determining accuracy, sensitivity, and specificity of the ePAT

Two steps were used to calculate these parameters. First, a 2×2 contingency table was set (Table 3). Next, sensitivity, specificity, accuracy, likelihood ratios, and predictive values were calculated (Table 4):

Table 4 Calculations of sensitivity, specificity, accuracy, likelihood ratios, and predictive values before prevalence adjustment Note: All values approximated to closest decimal point. Abbreviations: TP, true positive; FP, false positive; FN, false negative; TN, true negative.

sensitivity = (TP/[TP + FN]) ×100

= (295/[295+12]) ×100

= (295/307) ×100

= 96.1%

specificity = (TN/[TN + FP]) ×100

= (85/[85+8]) ×100

= (85/93) ×100

= 91.4%

accuracy = ([TP + TN]/[TP + TN + FP + FN]) ×100

= ([295+85]/[295+85+12+8])

= (380/400) ×100

= 95.0%

Clinical utility

CUI values were calculated using Mitchell’s formula.²¹ CUI+ is the CU of the test for case-finding (ie, confirmation), which is calculated as a product of sensitivity and positive predictive value, ie, CUI+ = 0.936 (95% CI 0.911–0.960).¹⁹ CUI^– is the CU of the test for screening (ie, ruling out pain), which is calculated as a product of specificity and negative predictive value, ie, CUI^– = 0.801 (95% CI 0.748–0.854).²¹ The overall ePAT value for combined screening and case finding was 95% (ie, CUI = 0.95). This meant the ePAT has excellent utility.²¹

Receiver-operating characteristic (ROC) curve

When the prevalence of pain adjusted to 50% (from 76.8% provided in Table 4), the sensitivity and specificity data were slightly varied. A choice of 50% prevalence was made to reflect the current data available in the literature.²² The optimal cutoff score where there was a balance of sensitivity and specificity was 7. This was confirmed by the area-under-the-curve value of 0.98 and P<0.0001. A Youden’s index value of close to 1 (ie, 0.88) pointed to minimal FPs and FNs in the data set. The closeness of the curve to the left corner of the graph illustrates ePAT’s high sensitivity and specificity in distinguishing between pain/no pain cutoffs. Results from the ROC curve used to analyze cutoff scores are presented in more detail in Table 5 and Figure 1.

Table 5 ROC curve characteristics, optimal criterion, and calculated sensitivity and specificity after prevalence adjustment Notes: aTaking into account disease prevalence (50%) and estimated costs: cost False Positive: 1; cost False Negative: 1 cost True Positive: 0; cost True Negative: 0 Positive group= pain present, Negative group= pain absent; bBCa bootstrap confidence interval (1000 iterations; random number seed: 978). Abbreviations: ePAT, electronic Pain Assessment Tool; ROC, receiver-operating characteristic; BCa, bias corrected and accelerated.

Figure 1 Electronic Pain Assessment Tool (ePAT) receiver-operating characteristic curve. Note: Data shown in the graph were based on pain prevalence of 50% (ie, after adjustment).

Discussion

This study evaluated the clinimetric properties (predictive validity and CU) of the ePAT in residents with moderate–severe dementia. For the ePAT application, this provides more evidence to how clinically meaningful the test results are in relation to pain detection in the target-population group. Other psychometric properties, such as concurrent validity, discriminant validity, and interrater reliability, have been reported elsewhere.^4,6

As reported by Herr et al⁹ and later by Lichtner et al,² currently the evidence for existing tools in regard to their validity and CU is limited and unclear. Due to these psychometric and clinimetric limitations, clinician and carer guidance on effective pain assessment is currently at best compromised, leading to gaps in relation to informed treatment options and care plans.² Some studies that have previously reported predictive validity of pain-assessment tools have been marred by failure actually to report the data, whereas others had significant scoring differences (pre- and postintervention).² The situation is similar in regard to the CU of pain-assessment tools. In addition to clinical data being completely or substantially absent for some tools, conflicting data have been reported for other tools.² Furthermore, CU dimensions, such as the cutoff scores (eg, pain vs no-pain cutoff described in the present study) needed for clinical decision making, have not been reported by many studies evaluating the currently available pain-assessment tools.² For example, the total score of the Pain Assessment in Advanced Dementia Scale can be 0–10 points. However, there is no evidence of pain-intensity scoring.²³ The original Pain Assessment Checklist for Seniors with Limited Ability to Communicate (PACSLAC) study did not report a specific cutoff score to determine the presence of pain. Rather, an increased PACSLAC score suggests that an increase in pain is likely, while a lower score suggests that pain has reduced.²⁴ Later, it was found that PACSLAC scores greater than 12 (of 60) are indicative of high pain intensity, whereas scores of 0–5 represent usual pain.²⁴ In the current study, the fact that the area under the curve was 0.98 (P<0.0001) indicated the ePAT’s strong ability to distinguish between cutoff scores for pain. Our current analysis confirmed a cutoff score of 7 for the presence of pain, also previously reported by Atee et al.⁴ This lends further support to our previous findings.⁴

This is the first study to demonstrate that the ePAT is a useful and valid instrument to assess pain in patients with moderate–severe dementia. This is an important step in further evidencing this instrument in the context of better assessment and management of pain in patients with dementia, given their compromised ability to communicate pain, which results in underrecognition and undertreatment of their pain.¹

High sensitivity and specificity values for the ePAT in identifying pain yield strong support to its predictive validity and responsiveness to change. Even after disease-adjusted prevalence analysis had been performed, sensitivity and specificity values changed only slightly. This validates our approach to the conceptual foundation of the tool, previously reported by Atee et al.⁴ Pain assessments were performed during rest and movement, and the resultant scores reflected the change in timing, regardless of the order of testing.⁶ The overall CUI value of the ePAT reported in this study was excellent (0.95) as indicated by Mitchell’s analysis. This suggests that the ePAT is a useful tool to assist in the clinical decision-making processes related to pain management, including informing clinicians on specific actions, such as analgesic administration, emphasized as a necessary property by van Herk et al¹³ when observing pain-assessment tools in patients with cognitive impairment. Currently, there are no published studies reporting Mitchell’s CUI of pain assessment tools in dementia. Other clinical tools where Mitchell’s CUI has been used include the Patient Health Questionnaire (PHQ9 and PHQ2) for depression in primary care, and the Cornell Scale for Depression in Dementia.^25,26 Our findings presented much better accuracy data than those tools.

Strengths and limitations

The present study has several points of strength. Statistical analyses were methodical and covered a number of variables. For example, Mitchell’s index was used for the first time (as far as we know) to investigate the CU of a pain-assessment tool for people with dementia. The data were gleaned from participants with varying types of dementia and pain conditions, and this also addresses recommendations by systematic reviews.^2,9 Full medical histories of residents were accessible to raters, which assisted in informing the clinical picture when doing pain assessments.

Among the limitations of our study (which was noted previously),⁶ it is important to highlight that the comparison was made with reference to the APS. Despite its frequent use in Australia, the APS is not a global gold standard. The original sample was relatively small, and contained disproportionate sex representation. Further, data obtained were limited to residential aged-care settings. Thus, the findings need to be interpreted with caution before applying them to other settings, eg, hospital. Also, there was no physical examination conducted for the sake of the study to identify the potential source of pain. In fact, pain data were based on medical history, where some aspects might not be a true representation of the current clinical status of residents. However, in the case of the ePAT, medical history was weighed as only 1 of the 42 points of the total scale.⁴ Therefore, it had a negligible effect on our data.

CU encompasses a wide range of effects that tests or tools can have on the patient.¹⁰ In this regard, it is worth emphasizing that a recent study conducted by van Kooten et al suggested that improving communication of results to physicians can lead to improvement in pain management in long-term care patients with dementia.²⁷ Although the present study did not evaluate this aspect, it is worth emphasizing that the ePAT tool is the only pain-assessment tool available that uses automation and is able to document and manage clinical information electronically.⁴ This is thus expected to have positive effects on pain detection and management and communication to physicians. This aspect needs to be further researched in future as an additional potential indicator of the ePAT’s clinical usefulness. CU also has a significant value for implementing tools in clinical practice. More emphasis and research on the CU of other pain-assessment tools are needed, because these parameters provide data on practicalities, which may improve the uptake and utilization of tools in clinical settings. Use of an electronic pain-assessment tool, which eases the process of documentation and pain tracking over time, has the potential to positively impact the tool’s CU through facilitating information on the temporality (ie, trends and patterns) of pain. In this regard, it is worth highlighting that the temporality of pain was regarded as “most useful to the assessment of pain” by Lichtner et al.²⁸ This potential strength of the ePAT application needs to be researched further in future.

Conclusion

The findings reveal strong clinimetric properties of the ePAT. We have demonstrated that both its predictive validity and CU are excellent, giving further evidence to the quality of pain assessment attained by the ePAT.

Acknowledgments

The authors express their gratitude to everyone involved in the study, including aged-care staff, residents, and their families. The authors would like to acknowledge the contribution of an Australian Government Research Training Program Scholarship in supporting this research. The original research that led to the development of the PainChek tool is part of a PhD project that was also supported by the Dementia Australia Research Foundation (DARF) through grant funding and a stipend scholarship. The content of the article is solely the responsibility of the authors, and does not necessarily represent the official views of DARF. The project has been commercialized initially through a startup company (ePAT Pty Ltd), and since October 2016 the Australian Share Securities (ASX)-listed EPAT Technologies Ltd (now known as PainChek Ltd). This research was also sponsored by EPAT Technologies Ltd.

Author contributions

KH, MA, and JDH conceived the idea and designed the study. KH and MA performed the literature review and drafted the manuscript. KH, MA, and JDH revised the manuscript. MA conducted and interpreted the statistical analyses. All authors contributed to and approved the final version of the manuscript.

Disclosure

KH, MA, and JDH are shareholders in PainChek Ltd (formerly known as EPAT Technologies), which is commercializing the ePAT instrument as PainChek^TM. They also have a patent application titled “A pain assessment method and system; PCT/AU2015/000501”, which has been under the national phase examination since February 2, 2017. KH is employed as a consultant by PainChek Ltd while serving as an assistant professor at the University of Prishtina and an adjunct senior lecturer at the School of Pharmacy, Curtin University. MA is a research scientist for PainChek Ltd while serving as a research fellow and PhD candidate with the School of Pharmacy, Curtin University. JDH is employed as chief scientific officer of PainChek Ltd while serving as a professor in the School of Pharmacy, Curtin University.

References

1.		Herr K, Coyne PJ, McCaffery M, Manworren R, Merkel S. Pain assessment in the patient unable to self-report: position statement with clinical practice recommendations. Pain Manag Nurs. 2011;12(4):230–250.
2.		Lichtner V, Dowding D, Esterhuizen P, et al. Pain assessment for people with dementia: a systematic review of systematic reviews of pain assessment tools. BMC Geriatr. 2014;14:138.
3.		Shega JW, Hougham GW, Stocking CB, Cox-Hayley D, Sachs GA. Pain in community-dwelling persons with dementia: frequency, intensity, and congruence between patient and caregiver report. J Pain Symptom Manage. 2004;28(6):585–592.
4.		Atee M, Hoti K, Parsons R, Hughes JD. Pain assessment in dementia: evaluation of a point-of-care technological solution. J Alzheimers Dis. 2017;60(1):137–150.
5.		Therapeutic Goods Administration eBusiness Services. ePAT Technologies Limited: information system software, application program. 2017. Available from: https://www.ebs.tga.gov.au/servlet/xmlmillr6?dbid=ebs/PublicHTML/pdfStore.nsf&docid=CD49D725FBC4131ECA2581610042344A&agid=(PrintDetailsPublic)&actionid=1. Accessed August 31, 2017.
6.		Atee M, Hoti K, Hughes JD. Psychometric evaluation of the electronic Pain Assessment Tool (ePAT): an innovative instrument for individuals with moderate to severe dementia. Dement Geriatr Cogn Disord. 2018;44(5–6):256–267.
7.		Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–549.
8.		Zwakhalen SM, Hamers JP, Abu-Saad HH, Berger MP. Pain in elderly people with severe dementia: a systematic review of behavioural pain assessment tools. BMC Geriatr. 2006;6:3.
9.		Herr K, Bjoro K, Decker S. Tools for assessment of pain in nonverbal older adults with dementia: a state-of-the-science review. J Pain Symptom Manage. 2006;31(2):170–192.
10.		Bossuyt PM, Reitsma JB, Linnet K, Moons KG. Beyond diagnostic accuracy: the clinical utility of diagnostic tests. Clin Chem. 2012;58(12):1636–1643.
11.		Abbey J, Piller N, de Bellis A, et al. The Abbey Pain Scale: a 1-minute numerical indicator for people with end-stage dementia. Int J Palliat Nurs. 2004;10(1):6–13.
12.		Gregersen M, Melin AS, Nygaard IS, Nielsen CH, Beedholm-Ebsen M. Reliability of the Danish Abbey Pain Scale in severely demented and non-communicative older patients. Int J Palliat Nurs. 2016;22(10):482–488.
13.		van Herk R, van Dijk M, Baar FP, Tibboel D, de Wit R. Observation scales for pain assessment in older adults with cognitive impairments or communication difficulties. Nurs Res. 2007;56(1):34–43.
14.		Salkind NJ. Encyclopedia of Research Design. Thousand Oaks (CA): SAGE; 2010.
15.		Cronbach LJ, Meehl PE. Construct validity for psychological tests. Psychol Bull. 1955;52(4):281–302.
16.		Jorm AF, Mackinnon AJ, Henderson AS, et al. The Psychogeriatric Assessment Scales: a multi-dimensional alternative to categorical diagnoses of dementia and depression in the elderly. Psychol Med. 1995;25(3):447–460.
17.		Hanley JA, Mcneil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36.
18.		Greiner M, Pfeiffer D, Smith RD. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med. 2000;45(1–2):23–41.
19.		Mitchell AJ. Sensitivity × PPV is a recognized test called the clinical utility index (CUI+). Eur J Epidemiol. 2011;26(3):251–252.
20.		Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem. 2003;49(1):7–18.
21.		Mitchell AJ. How do we know when a screening test is clinically useful? In: Mitchell AJ, Coyne JC, editors. Screening for Depression in Clinical Practice: An Evidence-Based Guide. Oxford: Oxford University Press; 2010:99–112.
22.		Achterberg WP, Pieper MJ, van Dalen-Kok AH, et al. Pain management in patients with dementia. Clin Interv Aging. 2013;8:1471–1482.
23.		Warden V, Hurley AC, Volicer L. Development and psychometric evaluation of the Pain Assessment in Advanced Dementia (PAINAD) scale. J Am Med Dir Assoc. 2003;4(1):9–15.
24.		Fuchs-Lacelle S, Hadjistavropoulos T, Lix L. Pain assessment as intervention: a study of older adults with severe dementia. Clin J Pain. 2008;24(8):697–707.
25.		Mitchell AJ, Yadegarfar M, Gill J, Stubbs B. Case finding and screening clinical utility of the Patient Health Questionnaire (PHQ-9 and PHQ-2) for depression in primary care: a diagnostic meta-analysis of 40 studies. BJPsych Open. 2016;2(2):127–138.
26.		Hancock P, Larner AJ. Cornell Scale for Depression in Dementia: clinical utility in a memory clinic. Int J Psychiatry Clin. 2015;19(1):71–74.
27.		van Kooten J, Smalbrugge M, van der Wouden J, Stek ML, Hertogh C. Evaluation of a pain assessment procedure in long-term care residents with pain and dementia. J Pain Symptom Manage. 2017;54(5):727–731.
28.		Lichtner V, Dowding D, Closs SJ. The relative meaning of absolute numbers: the case of pain intensity scores as decision support systems for pain management of patients with dementia. BMC Med Inform Decis Mak. 2015;15:111.