Children’s and adolescents’ relationship to pain during cancer treatment: a preliminary validation of the Pain Flexibility Scale for Children

Introduction

Children with cancer suffer from a number of symptoms throughout the cancer trajectory, of which pain is one of the most frequently reported and burdensome ones.¹ The children experience pain most often as a result of the disease itself, side effects of the cancer treatment, and/or procedures pertaining to the medical management.^2,3 The fact that pain is anxiety-provoking is well known,⁴ and children suffering from cancer are no exception.^5,6 Anxiety, in turn, amplifies the pain experience.^7,8 For persons with chronic pain, psychological acceptance has been shown to be helpful.^9–11 An acceptance-based psychological treatment, acceptance and commitment therapy, has been shown to improve psychosocial as well as physical functioning for both adults and children with chronic pain.^12,13 In acceptance and commitment therapy, the goal is to help people to engage in their lives in the presence of difficulties instead of being occupied with avoiding unpleasant stimuli. This is enabled by fostering psychological flexibility.¹⁴ Psychological acceptance is one of the key aspects of psychological flexibility. The definition of psychological acceptance of chronic pain is “living with pain without reacting to, judging or attempting to reduce or avoid it.”¹⁵ It entails actively engaging in meaningful life activities in the presence of pain, in order to continue to live life instead of putting it on hold, waiting for the pain to pass. Engaging in meaningful activities in the presence of aversive stimuli has been shown to precede a reduction in suffering from symptoms.¹⁶ For experimentally induced pain, acceptance-based interventions have been shown to predict an increase in pain tolerance and reductions in pain intensity and discomfort of pain.^17–21 In acceptance-based interventions, an attentive, nonreactive stance toward unpleasant stimuli is cultivated. The aim is to merely observe ongoing experiences without further mental evaluation.^22,23 This stance attenuates the pain experience and at the same time helps the person in pain to better choose his/her actions instead of rigidly reacting to internal and/or external events. A single subject study of an acceptance-based intervention for children reporting acute pain during cancer treatment has recently been undertaken at the Children’s University Hospital in Uppsala, Sweden (Cederberg, unpublished data, 2017). The aim of the intervention of the study was to help the children to practice a nonreactive stance toward painful stimuli with the purpose of helping them to cope with the pain and the emotional distress that the pain infers, thus giving them a means to continue to engage in their daily life activities in the presence of pain. All five participants reported decreased discomfort of pain postmeasurement. Psychological acceptance was the main treatment component, and hence the proposed mediator of the intervention. It is essential to evaluate and understand mechanisms of change in order to optimize treatments.²⁴ In the chronic pain area, several instruments measuring psychological acceptance have been reported.^25–27 In contrast, no instrument for measuring psychological acceptance for persons experiencing acute pain has yet been reported, least of all for children. With such an instrument at hand, the evaluation of psychological acceptance as a mediator for change in psychological interventions in the context of acute pain would be possible. This, in turn, would contribute to the development of acceptance-based psychological interventions that may help children with cancer experiencing pain to cope better with this challenging situation. The aim of this study was to develop and preliminarily evaluate an instrument measuring acceptance in the context of acute pain in children with cancer.

Methods

Participants and procedures

All children aged 7–18 years being treated for cancer in Sweden at the time of the study were invited to participate in the study. Two hundred thirty-three patients were identified by the Swedish Childhood Cancer Registry in November 2015. For one child, complete patient information was lacking, and he was therefore excluded. The research nurses at the six pediatric oncology centers in Sweden were consulted and double checked that the children had not gone into palliation or had died after the data had been extracted from the register. One child was identified as undergoing palliation and was therefore excluded. Two hundred thirty-one children were contacted in December 2015 via mail at their registered addresses. The study material consisted of information about the study, the test version of the scale, evaluation questions, and two measures for validation. The children were offered inclusion in a lottery of ten movie tickets on participation in the study. For the children, consent was given through participation in the study. In addition, a written parental consent was required for children under 15 years of age. Two weeks after the first dispatch, a reminder was sent out. One month after collection of the first measurement, the test material was sent out again for test–retest analysis. The study material contained no patient information, but was coded. The code key was kept in a locked space that could only be accessed by one of the researchers. Three dispatches were returned by the Postal Service. Sixty-two children (27%) participated in the study, of whom 39 participated at both measurements and 23 participated at one measurement, and one was excluded due to insufficient completion of the scales. Ten children declined participation. One hundred fifty-six children did not respond. The study was closed in May 2016. Table 1 provides a demographic overview. The study was approved by the Regional Ethical Review Board in Uppsala, Sweden [Dnr 2014/375].

Table 1 Gender, age, and diagnosis group of children Abbreviation: SD, standard deviation.

Background information

Background information included age, gender, type and date of diagnosis, and date of end of treatment (if applicable). Descriptive pain information included current level of pain and discomfort; highest, lowest, and average level of pain during the past week; average level of discomfort of pain during the past week; and type of pain. Pain and level of discomfort was rated on a scale from 0 = “No pain/discomfort at all” to 10 = “Unbearably lot of pain/discomfort.”²⁸

Development of the pain flexibility scale for children

Three psychologists theoretically and clinically familiar with the concept of acceptance were involved in the development of the Pain Flexibility Scale for Children (PFS-C). First, a draft of different potential dimensions of acceptance was elaborated. Second, the Chronic Pain Acceptance Questionnaire (CPAQ) was used as a basis for the new scale.^25,29 The CPAQ is designed to measure acceptance in patients with chronic pain and contains 20 items divided into two subscales. The Activity Engagement subscale measures engagement in meaningful activities in spite of the presence of pain, and the Pain Willingness subscale measures the degree to which the respondent tries to avoid or control pain. A higher score indicates a higher level of acceptance. Internal consistency was shown to be α =0.78–0.82. The CPAQ correlates negatively with measures of physical and psychosocial disability. The same response format was used; a seven-point Likert scale. The scale ranged from 0 = “Completely disagree (Never true)” to 6 = “Entirely agree (Always true).” Eleven items from the CPAQ that were clearly chronic pain oriented (#2, #4, #5, #6, #9, #10, #12, #13, #14, #18, and #19), such as “My life is going well, even though I have chronic pain,” were deleted. Nine items from the CPAQ (#1, #3, #7, #8, #11, #15, #16, #17, and #20) were retained. These were reframed to suit the process of acceptance in the context of acute pain in children. For example, Item 11 “My thoughts and feelings about pain must change before I can take important steps in life” was reframed to “The pain needs to pass before I can focus on anything else.” Third, 29 new items were generated in accordance with the draft of potential dimensions of acceptance in the context of acute pain. The language was adapted to suit children. Two children, aged 8 and 10 years, filled in the test scale to assess the appropriateness of the level of language. No adjustments were called upon based on their feedback. The final test version contained 38 items. In order to synchronize the direction of the scale, twenty-three items reflecting the opposite pole of the dimension, such as “Being in pain is too difficult for me” and “I need to focus on getting rid of the pain,” were reversed before performing the statistical analyses.

Measures used for validation

Two measures were used to evaluate convergent validity. The Pain Catastrophizing Scale for Children (PCS-C) is designed to measure catastrophizing thoughts in children in pain.^30,31 The scale consists of 13 statements with which the children rate their agreement on a five-point Likert scale. Examples of statements are: “When I have pain, I worry all the time about whether the pain will end” and “When I have pain, I get scared that the pain will get worse.” The score range is 0–52, and a higher score indicates a higher level of catastrophizing. The PCS-C correlates with measures of depressed mood and trait anxiety. Internal consistency has been shown to be good (α =0.87). The Avoidance and Fusion Questionnaire for Youth (AFQ-Y) is designed to measure psychological inflexibility in youths.^32,33 Respondents rate to what extent they agree with statements targeting experiential avoidance and cognitive fusion such as “My life won’t be good until I feel happy” and “I am afraid of my feelings.” The response format is a five-point Likert scale. The score range is 0–32, and a higher score indicates a higher level of psychological inflexibility. The short version of eight items was used, which correlates positively with child-reported anxiety, physical symptoms, and problem behavior and negatively with general quality of life. Internal consistency has been shown to be good (α =0.83).

Statistical analyses

Initial analyses of the test version of the scale were carried out to assess the suitability of factor analysis. Internal consistency was calculated, frequency distributions were examined, and inter-item and item–total correlations were inspected. Preliminary factor analysis was performed whereby eigenvalue, scree plot, and pattern matrices were evaluated to select the number of factors to retain for final factor analysis. Principal component analysis was used. Internal consistency and test–retest reliability was calculated for the final total scale and the subscales. The intraclass correlation coefficient was used to calculate test–retest reliability where a two-way random effects model using an absolute agreement definition was applied.³⁴ The Single Measures value was assessed.³⁵ An ICC of less than 0.40 indicates poor agreement, between 0.40 and 0.59 fair, between 0.60 and 0.74 good, and more than 0.75 excellent.³⁶ Correlations with other measures were carried out in order to assess convergent validity. The data on all scales was normally distributed, and Pearson correlation was used. Correlation coefficients were interpreted according to the guidelines recommended by Cohen³⁷ (r=0.10–0.29 small, 0.30–0.49 medium, and 0.5–1.0 large). Level of statistical significance was set at p<0.05. All statistical analyses were performed in IBM SPSS Statistics, version 24 (Armonk, NY, USA).³⁸

Results

Descriptives

Sixty-one children participated in the study. Reports of level of pain and discomfort are presented in Table 2, and reports of type of pain are presented in Table 3.

Table 2 Reports of level of pain and discomfort Abbreviation: SD, standard deviation.

Table 3 Reports of type of pain

Factor analysis

Cronbach’s α for the test version of the scale was 0.78, and hence internal consistency was acceptable. Frequency distributions showed that the data on some items were skewed, which was expected. The variability was however considered acceptable for all items. The data were normally distributed for the total test scale. Outliers were identified on items 1 and 2 and on the total scale. The outliers had very little effect on the mean and were retained in the analyses. Eleven items had corrected item–total correlations below zero and were eliminated from further analysis. They were as follows: Item 1, “I prepare to fight when I get pain”; Item 2, “Even though it is difficult for me to be in pain, I know that I can handle it”; Item 3, “I refuse to feel the pain”; Item 12, “If I think about something else I can handle being in pain”; Item 20, “The pain gets easier if I try to control it”; Item 22, “How I react when I get pain is different from one time to another”; Item 23, “If I grit my teeth I can stand being in pain”; Item 26, “Sometimes it is unavoidable to have pain”; Item 33, “If I try to feel what I really actually feel, it is easier,” ”; Item 36, “Sometimes I am actually curious about the pain”; and Item 37, “The pain gets worse if I try to control it.” After the elimination of these eleven items, five items had item–total correlations below 0.3 and were eliminated. They were as follows: Item 4, “Sometimes it feels OK to experience pain”; Item 14, “Even though it is difficult to be in pain I have learned that I can actually handle it”; Item 24, “I need to control the pain”; Item 32, “I try to help myself cope with the pain”; and Item 35, “I do things to flee from the pain.” Furthermore, after elimination of these five items, the item–total correlation for Item 19 “Sometimes I feel that I am greater than the pain” had sunk to below 0.3, and consequently Item 19 was eliminated. This, in turn, lowered the item–total correlation for Item 34 “If I try to feel what I really actually feel, it is more difficult” to below 0.3, and therefore Item 34 was also eliminated. Principal component analysis was performed on the remaining 20 items. Preliminary factor analyses showed no items loading independently of the others. Bartlett’s test of sphericity was significant and the Kaiser–Meyer–Olkin Index was 0.76. Interdependence between factors was indicated, and oblique rotation was used. From the preliminary factor analyses, six factors were extracted with eigenvalues above 1, while the scree plot indicated two factors to retain. The component and pattern matrices supported a two-factor solution, which was chosen. All items had factor loadings above 0.4 and communalities above 0.3. Twenty items were included in the final solution, and Promax was chosen as the rotation method. Variance explained by the factor solution was 54%: 37% by the first and 17% by the second factor. Table 4 provides the final factor solution. The theoretical analysis of the item content of the factors brought forth the following factor labels: 1) Valued Action and 2) Pain Resistance. The first factor, Valued Action, is about continuing to live in the presence of pain instead of being occupied with trying to control pain. It is also characterized by a nonevaluative perspective of pain in relation to one’s ability to cope with it. The second factor, Pain Resistance, is about resisting and trying to control pain and/or the feelings that being in pain infers. It is also characterized by a kind of reactivity to pain as pain is seen as threatening and unmanageable. The score range is 0–120 for the total scale, 0–54 for the Valued Action subscale, and 0–66 for the Pain Resistance subscale.

Table 4 Factors, items, factor loadings, and communalities for the final solution (n=61)

Reliability and validity

Scale characteristics and reliability and validity coefficients are presented in Table 5. The test–retest correlation coefficients indicated good agreement for the total scale and fair agreement for the subscales. Controlling for change in level of pain had a negligible effect on these correlations. The correlations with the PCS-C were large for the total scale and the Valued Action subscale and medium for the Pain Resistance subscale. When controlling for level of pain, the correlations between the PCS-C and the total scale and the Valued Action subscale were unchanged, but the correlation between the PCS-C and Pain Resistance subscale changed from −0.43 to −0.41. The correlations with the AFQ-Y were medium for the total scale and the Pain Resistance subscale and small for the Valued Action subscale. When controlling for level of pain, the correlation between the AFQ-Y and the Valued Action subscale was unchanged, but the correlation between the AFQ-Y and the total scale changed from −0.36 to −0.33 and correlation with the Pain Resistance subscale from −0.32 to −0.28. Hence, for the Pain Resistance subscale, controlling for level of pain changed the effect from medium to small. Regarding all other effects of level of pain on the correlations, these did not change the interpretation of the strength of the correlation. All correlations were significant (p<0.05).

Table 5 Mean, SD, score range, internal consistency, and correlation coefficients for the total scale and the subscales Abbreviations: PCS-C, The Pain Catastrophizing Scale for Children; AFQ-Y, The Avoidance and Fusion Questionnaire for Youth; SD, standard deviation.

Discussion

The aim of the current study was to develop and preliminarily evaluate an instrument for measuring acceptance in the context of acute pain in children. This would enable the investigation of acceptance as a mediator for change in acceptance-based interventions that may help children with cancer experiencing pain to cope better. Factor analysis was used, and a two-factor solution was chosen. The final scale, the PFS-C, consisted of 20 items. The two subscales were Valued Action and Pain Resistance. Regarding the name of the scale, the term “Flexibility” was chosen instead of “Acceptance” to indicate the theoretically slightly broader scope of the scale, including the Valued Action subscale. The total scale and the Valued Action subscale showed excellent internal consistency, while the Pain Resistance subscale showed good internal consistency. Furthermore, the PFS-C demonstrated temporal stability and satisfactory convergent validity.

The sample of the study was small, especially considering the statistical method used, ie, factor analysis. It is often a challenge to achieve large enough samples in clinical studies in general, and in pediatric clinical studies in particular. A consequence of this challenge is that research in pediatric clinical settings runs the risk of being overlooked and thus not being conducted. The population of 231 children was an in-built limitation. However, given the significance of the prevailing of pediatric clinical research in spite of the challenge of small populations and the importance of the development and evaluation of instruments enabling investigation of mediators for treatment change in order to optimize interventions for children in pain, the study was considered important despite this limitation. Almost a third of the children participated in the study. Considering the format of the study and the often intense situation that undergoing cancer treatment implies for these children, this response frequency must be deemed good enough under the circumstances. This should however be kept in mind when generalizing the results of the study. Some respondents communicated that the questions were difficult to understand. Given the nature of the questions, this was expected and considered inevitable to some extent. The respondents were evenly distributed across the whole age span, ranging from 7 to 18 years, showing that younger children participated to the same extent as older ones. Yet, the possibility of children not participating in the study due to perceived difficulty is, also, something to keep in mind when generalizing the results. All children aged 7–18 years undergoing cancer treatment at the time of the study were invited to participate in the study. Respondents were asked to rate their level of pain. Many respondents had previously experienced pain but were not in pain at the time of the measurement. For those children, the measurements were completed retroactively. This may be the reason why the reported level of pain is as low as it is. Even though retroactive measurements are not desirable, taking into consideration the likelihood that experiencing pain is a strongly unpleasant experience for a child and that the pain episode is likely to have occurred relatively recently, these ratings were considered to be valid. Information about previous experience of pain and its time frames was not collected. In the absence of such background questions, there was a risk of collecting data from children who had not experienced any pain during cancer treatment. Previous research¹ and clinical experience suggest however that this would be unlikely. Furthermore, several respondents commented that they referred to a previous pain episode when filling in the scale. The children who explicitly declined participation in the study often stated that their pain had been very limited in time, for example as a side effect of surgery. The risk of including children who had not experienced any pain associated with their cancer or cancer treatment is therefore considered small.

The study is a preliminary validation of the PFS-C. Further validation is always important in the development of new scales, particularly if they are to be used for other populations. In this case, this is especially important, considering the small sample size of the study. A Swedish version of the scale has been developed and evaluated. To be used as an English version, it needs to be validated first. Sensitivity to change also needs to be assessed.

In summary, a scale for measuring acceptance of acute pain in children with cancer is now available for use, enabling the investigation of acceptance as a mechanism of treatment change in this context. This is a step forward in the development of acceptance-based psychological interventions that may help children and adolescents with cancer to cope better with the pain that is often associated with cancer treatment. Given the small sample size of the study, the results should be seen as tentative.

Acknowledgments

This work was supported by grants by the Swedish Childhood Cancer Foundation [FTJH11/002 & PR2013/0058], the Swedish Cancer Society [CAN2013/749], and the Gillbergska Foundation. We would like to thank the Swedish Childhood Cancer Registry for identifying potential participants for our study and providing us with data. The research nurses of the six pediatric oncology centers in Sweden are also gratefully acknowledged for screening the participant list in order to ensure that no child who had recently gone into palliation or had died would be contacted. Further, we would also like to acknowledge the developers of the CPAQ, upon which the test version of the PFS-C was based.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Twycross A, Parker R, Williams A, Gibson F. Cancer-related pain and pain management: sources, prevalence, and the experiences of children and parents. J Pediatr Oncol Nurs. 2015;32(6):369–384.
2.		International Association for the Study of Pain (IASP). 2008–2009 Global Year Against Cancer Pain. [IASP Web site]. Available from: http://www.iasp-pain.org/GlobalYear/CancerPain. Accessed March 8, 2017.
3.		Ljungman G, Kreuger A, Gordh T, Berg T, Sörensen S, Rawal N. Treatment of pain in pediatric oncology: A Swedish nationwide survey. Pain. 1996;68(2–3):385–394.
4.		Linton SJ, Shaw WS. Impact of psychological factors in the experience of pain. Phys Ther. 2011;91(5):700–711.
5.		Hedström M, Haglund K, Skolin I, von Essen L. Distressing events for children and adolescents with cancer: child, parent, and nurse perceptions. J Pediatr Oncol Nurs. 2003;20(3):120–132.
6.		Hedén L, Pöder U, von Essen L, Ljungman G. Parents’ perceptions of their child’s symptom burden during and after cancer treatment. J Pain Symptom Manage. 2013;46(3):366–375.
7.		Cioffi I, Michelotti A, Perrotta S, Chiodini P. Effect of somatosensory amplification and trait anxiety on experimentally induced orthodontic pain. Eur J Oral Sci. 2016;124(2):127–134.
8.		Pagé GM, Campbell F, Isaac L, Stinson J. Reliability and validity of the Child Pain Anxiety Symptoms Scale (CPASS) in a clinical sample of children and adolescents with acute postsurgical pain. Pain. 2011;152(9):1958–1965.
9.		McCracken LM, Gutiérrez-Martínez O. Processes of change in psychological flexibility in an interdisciplinary group-based treatment for chronic pain based on Acceptance and Commitment Therapy. Behav Res Ther. 2011;49(4):267–274.
10.		Vowles K, Witkiewitz K, Sowden G, Ashworth J. Acceptance and commitment therapy for chronic pain: evidence of mediation and clinically significant change following an abbreviated interdisciplinary program of rehabilitation. J Pain. 2014;15(1):101–113.
11.		Thorsell Cederberg J, Cernvall M, Dahl J, von Essen L, Ljungman G. Acceptance as a mediator for change in acceptance and commitment therapy for persons with chronic pain? Int J Behav Med. 2016;23(1):21–29.
12.		Veehof MM, Oskam MJ, Schreurs KM, Bohlmeijer ET. Acceptance-based interventions for the treatment of chronic pain: a systematic review and meta-analysis. Pain. 2011;152(3):533–542.
13.		American Psychological Association (APA). Evidence of Psychological Treatments. [APA Web site]. Available from: http://www.div12.org/psychological-treatments/disorders/chronic-or-persistent-pain/acceptance-and-commitment-therapy-for-chronic-pain/. Accessed March 8, 2017.
14.		Hayes S, Strosahl K, Wilson K. Acceptance and Commitment Therapy: The Process and Practice of Mindful Change. 2nd ed. New York, NY: The Guilford Press; 2012.
15.		McCracken L. Learning to live with the pain: acceptance of pain predicts adjustment in persons with chronic pain. Pain. 1998;74(1):21–27.
16.		Gloster AT, Klotsche J, Ciarrochi J, et al. Increasing valued behaviors precedes reduction in suffering: findings from a randomized controlled trial using ACT. Behav Res Ther. 2017;91:64–71.
17.		Zeidan F, Gordon N, Merchant J, Goolkasian P. The effects of brief mindfulness meditation training on experimentally induced pain. J Pain. 2010;11(3):199–209.
18.		Zeidan F, Martucci K, Kraft R, Gordon N, McHaffie J, Coghill R. Brain mechanisms supporting the modulation of pain by mindfulness meditation. J Neurosci. 2011;31(14):5540–5448.
19.		Brown C, Jones A. Meditation experience predicts less negative appraisal of pain: Electrophysiological evidence for the involvement of anticipatory neural responses. Pain. 2011;150(3):428–438.
20.		Choi K, Rampp T, Saha F, Dobos G, Musial F. Pain modulation by meditation and electroacupuncture in experimental submaximum effort tourniquet technique (SETT). Explore. 2011;7(4):239–245.
21.		Forsyth L, Hayes LL. The effects of acceptance of thoughts, mindful awareness of breathing, and spontaneous coping on an experimentally induced pain task. Psychol Rec. 2014;64(3):447–455.
22.		Keogh E, Bond FW, Hanmer R, Tilston J. Comparing acceptance- and control-based coping instructions on the cold-pressor pain experiences of healthy men and women. Eur J Pain. 2005;9(5):591–598.
23.		Buhle J, Wager TD. Does meditation training lead to enduring changes in the anticipation and experience of pain? Pain. 2010;150(3):382–383.
24.		Kazdin AE, Nock MK. Delineating mechanisms of change in child and adolescent therapy: methodological issues and research recommendations. J Child Psychol Psyc. 2003;44(8):1116–1129.
25.		McCracken LM, Vowles KE, Eccleston C. Acceptance of chronic pain: component analysis and a revised assessment method. Pain. 2004;107(1):159–166.
26.		Wicksell RK, Lekander M, Sorjonen K, Olsson GL. The Psychological Inflexibility in Pain Scale (PIPS) – Statistical properties and model fit of an instrument to assess change processes in pain related disability. Eur J Pain. 2010;14(7):771.e1–e14.
27.		Reneman MF, Kleen M, Trompetter HR, et al. Measuring avoidance of pain: validation of the Acceptance and Action Questionnaire II-pain version. Int J Rehabil Res. 2014;37(2):125–129.
28.		Ferreira-Valente MA, Pais-Ribeiro JL, Jensen MP. Validity of four pain intensity rating scales. Pain. 2011;152(10):2399–2404.
29.		Vowles KE, McCracken LM, McLeod C, Eccleston C. The chronic pain acceptance questionnaire: confirmatory factor analysis and identification of patient subgroups. Pain. 2008;140(2):284–291.
30.		Sullivan MJ, Bishop SR, Pivik J. The pain catastrophizing scale: development and validation. Psychol Assess. 1995;7(4):524–532.
31.		Crombez G, Bijttebier P, Eccleston C, et al. The child version of the pain catastrophizing scale (PSC-C): a preliminary validation. Pain. 2003;104(3):639–646.
32.		Greco LA, Lambert W, Baer RA. Psychological inflexibility in childhood and adolescence: development and evaluation of the Avoidance and Fusion Questionnaire for Youth. Psychol Assessment. 2008;20(2):93–102.
33.		Livheim F, Tengström A, Bond FW, Andersson G, Dahl J, Rosendahl I. Psychometric properties of the Avoidance and Fusion Questionnaire for Youth: A psychological measure of psychological inflexibility in youth. J Contextual Behav Sci. 2016;5(2):103–110.
34.		Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42.
35.		Weir JP. Quantifying Test-retest Reliability using the Intraclass Correlation Coefficient and the SEM. J Strength Cond Res. 2005;19(1):231–240.
36.		Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol assessment. 1994;6(4):284–290.
37.		Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale: L. Erlbaum Associates; 1988.
38.		IBM. SPSS Statistics. Version 24.0.0. Armonk, NY: IBM Corporation; 2016.