The Oxford Participation and Activities Questionnaire: study protocol

Introduction

With an ageing population and increasing demands on health and social care services, growing importance is attached to the management of long-term conditions, including maximizing the cost-effectiveness of treatments. In line with this, there is increasing emphasis on the need to keep people both active and participating in daily life.^1–4 Consequently, it is essential that well developed and validated instruments that can meaningfully assess levels of participation and activity are widely available.

The current literature, however, demonstrates little consensus on how we define “participation” and “activity”. A number of papers have been published with the aim of clarifying or reaching agreement on a clear definition for both terms,^5–7 but with limited success. Some have called for greater clarity between the two concepts,⁸ whilst others have concluded that participation is a complex, elusive, and subjective concept, influenced by one’s individual perspective.⁹ This serves to highlight the importance of investigating the meaning of concepts such as participation and activity with those in whom we are trying to measure it. As previous qualitative research demonstrates, when this is undertaken, the complex nature of what, for example, participation actually means to different individuals becomes evident.¹⁰

One widely used definition of the concepts of participation and activity is to be found in the World Health Organization International Classification of Functioning, Disability and Health (ICF),¹¹ which at least provides a framework within which to construct scales aimed at measuring the two constructs. The ICF defines participation as “involvement in life situations” and activity as “the execution of a task or action by an individual”. Although initially described as two separate concepts, the final version of the ICF merges activity and participation into a single taxonomy and identifies nine domains which are presented in Table 1.

Table 1 Domains of the ICF classification of participation and activity Abbreviation: ICF, International Classification of Functioning, Disability and Health.

There is still significant debate over the ICF model and particularly in relation to whether activity and participation is a single entity, or whether a clear distinction should be made between the two.^12,13 Despite this continuing debate, a number of patient-reported measures of activity and participation have now been constructed, to varying degrees, based on the model.¹⁴ Recently published reviews and clinical commentaries, however, highlight significant limitations in a number of these measures.^15–20 Current measures are largely focused on disability and rehabilitation, with many falling short of the full range of psychometric properties required for patient-reported outcome measures (PROMs) under guidance such as that provided by the US Food and Drug Administration (FDA).²¹ The development of a number of instruments is poorly reported and some have been developed with small sample sizes, which calls into question the legitimacy of the psychometric analyses undertaken. Many have failed to involve patients in generating items, something that is considered fundamental in developing relevant PROMs that accurately reflect the concerns of those who are being assessed.²¹

Such a picture provides a sound rationale for developing a new measure, the Oxford Participation and Activities Questionnaire (Ox-PAQ). The Ox-PAQ initiative aims to address a number of the issues highlighted above by developing a measure theoretically grounded in the ICF for generic use and with patients at the heart of the item-generation process. Additionally, the views of stakeholders will be gained from the outset in an attempt to identify those characteristics of PROMs considered most important to a range of potential users. The measure will subsequently be assessed for the full range of psychometric properties, and sample sizes will allow for detailed and legitimate psychometric analyses. The development process will be entirely in line with FDA guidelines which are now considered best practice, regardless of whether an instrument is being incorporated into, or developed for, trials supporting pharmaceutical label claims.²²

Materials and methods

Ethical approval for the study was granted by the Medical Sciences Inter-Divisional Research Ethics Committee of the University of Oxford (reference MSD-IDREC-C1-2013-064). The study will follow a nine-point development process, as outlined in Table 2. The methodology (participants, materials, procedure, and statistical analysis) as to how this development process will be achieved is detailed below.

Table 2 Developmental stages of the Ox-PAQ initiative Abbreviation: Ox-PAQ, Oxford Participation and Activities Questionnaire.

Participants

Development of the Ox-PAQ will require three separate groups of participants. The source of recruitment and nature of participants will depend on the particular stage of the project as outlined below.

Stakeholder interviews

A broad range of participants has been drawn from a number of areas where patient-reported outcomes are of relevance (ie, medicine, academia, regulation).

Patient interviews

A sample of patients (approximately 30–40) drawn from a wide range of conditions, including arthritis, cancer, chronic back pain, diabetes, motor neurone disease, multiple sclerosis, Parkinson’s disease, and spinal cord injury, will be recruited via relevant support charities. The patient groups included represent a spectrum of conditions that together affect a number of key bodily systems, as well as having different symptoms, trajectories, and prognoses.

Survey participants

A large separate sample of patients with the conditions outlined above will again be recruited via relevant support charities.

Inclusion/exclusion criteria

Participation in the stakeholder interviews requires participants to have some knowledge and previous experience with PROMs, and be actively engaged in a field where PROMs are of relevance. Inclusion in the patient interview phase will require participants to have a confirmed diagnosis of the condition being investigated. Survey participants will be excluded if they are unable to complete the questionnaires independently and if they do not have English as a primary language. All participants must be aged 18 years or over.

Materials

Two instruments will be administered alongside the Ox-PAQ for the purpose of evaluating its validity.

The SF-36

The SF-36^23,24 is a 36-item instrument containing eight domains (physical functioning, role physical, role emotional, social functioning, mental health, energy/vitality, pain, and general health perception) that has been extensively used and shown to have excellent psychometric properties.^25,26 Raw scores for each domain are transformed to have a range from 0 to 100, with higher scores indicative of superior health status. Two summary scores can also be calculated from the results of the eight domains, ie, the Physical Component Summary and Mental Component Summary.^27,28

The EQ-5D

The EQ-5D²⁹ is a generic, five-item questionnaire that also incorporates a visual analog scale (ranging from 0 [worse health] to 100 [best health]) to indicate general health status. A unique health state is derived by combining responses from each of the five questions. Each score is intended to reflect societal valuations of the health state in terms of “utility” or valuations of health-related quality of life. Any one of the possible 243 health states that can be derived is converted to a summary score, with higher scores indicating superior quality of life. The instrument has acceptable reliability for both the utility (r=0.73) and visual analog scale (r=0.70) components.³⁰

Procedure

The procedure for the nine developmental stages outlined in Table 2 will be conducted as follows.

Semistructured interviews have been conducted across Europe with 18 professionals from a range of different backgrounds and areas of expertise. These have included regulation, commissioning, reimbursement, clinical practice, nursing, health economics, national government, academia, clinical trials, patient support, and health technology assessment. The purpose of these interviews was two-fold: firstly, to discuss issues surrounding current PROMs and, secondly, to consider the broad topic areas a generic measure of participation and activity might include that will be meaningful and considered important. No previous measure has been designed from the outset with the views of such a variety of potential users at its core. Furthermore, the interviews discussed the practicalities of the instrument, such as preferred methods of scoring and rules for interpretation. The interviews will be transcribed, subjected to content analysis, and used to inform later stages of the project.

Exploratory, open-ended, face-to-face, indepth interviews will be undertaken with a sample of patients across a wide range of conditions. The interviews will aim to identify salient dimensions of illness impact that have adversely affected participation and activities. It is anticipated that approximately 30–40 patients will be interviewed in depth until, as recommended, “no new themes emerge”.³¹ Interviews will be recorded, transcribed, and subjected to content analysis. Whilst there will be an interview guide, participants will be free to range across this and address any other relevant topics.

The interviews conducted in Stage 2 will be transcribed and transcripts will be scrutinized independently by the research team for issues relating to illness impact. Issues arising will be recast as questionnaire items to be answered on a five-point Likert scale. A Flesch readability score³² will be calculated for each item with a target of approximately grade 6. A meeting of health care researchers and relevant interested stakeholders will be convened to assess the list of items for completeness, ambiguity, and repetition. The resulting candidate questionnaire will then be discussed with a small group of people representing a range of conditions in a focus group setting for their comments and to assess its face validity. Additionally, the questionnaire will be pretested with a variety of patients in cognitive interviews using the format outlined by Willis.³³ Participants will be asked to complete the questionnaire and explain the reasons for their answers to each question, and comment upon any difficulties or ambiguities they experience.

The long-form questionnaire drawn up in Stage 3 will be mailed to a sample of 600 people with a variety of long-term conditions. This sample size is based on the assumption that the instrument will contain approximately 50 items. Estimates suggest that a minimum of five times as many respondents than items are required for psychometric tests to be meaningful.³⁴ Assuming a typical response rate of between 60% and 70%,³⁵ this will lead to a sample of approximately 360–420 questionnaires, which will permit rigorous testing of the instrument.

Analysis of data from Stage 4 will allow for identification of a shorter form questionnaire using established statistical procedures as detailed below in the Statistical analysis section. Items with high face validity, good correlation with the scale total to which they contribute, and sound reliability will constitute a dimension of the resulting instrument.

The short-form measure generated in Stage 5, together with the SF-36^23,24 and EQ-5D,²⁹ will be mailed to 1,000 patients with a wide variety of conditions. Assuming a 60%–70% response rate (as discussed previously), this will yield a final sample size of approximately 600–700 returned questionnaires.

Further statistical analyses will be undertaken on data from Stage 6 to confirm the results from the initial survey in Stage 5. Minor amendments may be made to the questionnaire at this time.

Previous respondents will be asked to complete the questionnaire once more. If they agree to do so, they will be sent the Ox-PAQ one week after receipt of the measure from Stage 6 in order to assess test-retest reliability. Previous respondents will be asked to complete the questionnaire for the last time after 3 months has elapsed in order to assess the instrument’s responsiveness.

Statistical analysis

Data will be checked for presence of outliers and normality of distribution prior to statistical analysis. Distribution of the data will be analyzed using the Shapiro-Wilk test of normality.³⁶ Subscale and summary scores for the SF-36 and EQ-5D will be calculated according to respective scoring algorithms. Items of the Ox-PAQ will be assessed for floor and ceiling effects under the criteria outlined by Terwee et al.³⁷ Factor analytic techniques will subsequently be performed to assess construct validity and identify dimensions relating to specific aspects of participation and activity. Item-total correlations, corrected for overlap, will be calculated between items and the total score to which they contribute. Rasch analysis^38–40 will also be conducted to determine which items conform to a hierarchical unidimensional structure. Further tests of validity (concurrent and discriminant) will be measured by calculating Pearson’s product-moment⁴¹ or Spearman rank correlation⁴² coefficients (as appropriate depending on data distribution) between the Ox-PAQ, SF-36, and EQ5D. Reliability will be assessed via tests of internal consistency using Cronbach’s alpha coefficient⁴³ and test-retest reliability using intraclass correlations.⁴⁴ Responsiveness to change will be determined through calculation of standardized effect sizes.⁴⁵ Data will be coded in order to maintain confidentiality and analysed using Statistical Package for the Social Sciences version 20 software (IBM, Armonk, NY, USA). Rasch analysis will be conducted using RUMM2030.

Discussion

This paper has described the initial stages of developing and validating the Ox-PAQ, a new measure of participation and activity. The rationale for developing this instrument is relatively straightforward. Current measures are largely focused on disability and rehabilitation, and there is no measure of activity and participation for generic use that fully meets the standards set by regulatory bodies such as the FDA. The Ox-PAQ initiative aims to ameliorate this. The measure will be broadly focused and for use in a range of conditions. The developmental stages discussed follow current guidelines and represent best practice in the development of outcome measures. Additionally, the project incorporates a new strategy of involving and engaging with potential users from the outset of the development process.

The development of the Ox-PAQ is a timely one. With increasing emphasis being placed on the importance of keeping people active and participating in daily life, the instrument has the potential for significant uptake. Its primary use is intended to be in clinical trials and related forms of evaluation of interventions targeted at maintaining activity and participation. Given the increasing importance of electronic data capture, the development of an electronic version of the Ox-PAQ, for use on platforms such as computers, tablets, and smartphones, is something that may follow its initial development and validation. The progress of the development and validation of the Ox-PAQ can be followed at the University of Oxford Health Services Research Unit website (http://www.dph.ox.ac.uk/research/hsru/OxPAQ), and results will be disseminated via international conferences and journals.

Acknowledgment

The Ox-PAQ initiative is funded by the European Brain Council. We would like to acknowledge the continued support and assistance of Dr Mary Baker, President, European Brain Council.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.	Macfarlane J. Other physical consequences of disability. Handb Clin Neurol. 2013;110:315–322.
2.	Blickem C, Kennedy A, Vassilev I, et al. Linking people with long-term health conditions to healthy community activities: development of Patient-Led Assessment for Network Support (PLANS). Health Expect. 2013;16:e48–e59.
3.	Beer S, Khan F, Kesselring J. Rehabilitation interventions in multiple sclerosis: an overview. J Neurol. 2012;259:1994–2008.
4.	Provencher V, Bier N, Audet T, Gagnon L. [Long-term effect of a cognitive intervention on learning and participation in a significant leisure activity in early dementia of Alzheimer type: a case study]. Psychol Neuropsychiatr Vieil. 2009;7:131–140. French.
5.	Dijkers MP. Issues in the conceptualization and measurement of participation: an overview. Arch Phys Med Rehabil. 2010;91 Suppl 1:S5–S16.
6.	Heinemann AW, Tulsky D, Dijkers M, et al. Issues in participation measurement in research and clinical applications. Arch Phys Med Rehabil. 2010;91 Suppl 1:S72–S76.
7.	Whiteneck GG, Bogner JA, Heinemann AW. Advancing the measurement of participation. Arch Phys Med Rehabil. 2011;92:540–541.
8.	Badley EM. Enhancing the conceptual clarity of the activity and participation components of the International Classification of Functioning, Disability, and Health. Soc Sci Med. 2008;66:2335–2345.
9.	Magasi S, Hammel J, Heinemann A, Whiteneck G, Bogner J. Participation: a comparative analysis of multiple rehabilitation stakeholders’ perspectives. J Rehabil Med. 2009;41:936–944.
10.	Hammel J, Magasi S, Heinemann A, Whiteneck G, Bogner J, Rodriguez E. What does participation mean? An insider perspective from people with disabilities. Disabil Rehabil. 2008;30:1445–1460.
11.	World Health Organization. International Classification of Functioning, Disability and Health. Geneva, Switzerland: World Health Organization; 2001. Available from: http://www.who.int/classifications/icf/en/. Accessed October 31, 2013.
12.	van der Zee CH, Kap A, Rambaran Mishre R, Schouten EJ, Post MW. Responsiveness of four participation measures to changes during and after outpatient rehabilitation. J Rehabil Med. 2011;43:1003–1009.
13.	Whiteneck CG, Dijkers MP. Difficult to measure constructs: conceptual and methodological issues concerning participation and environmental factors. Arch Phys Med Rehabil. 2009;90(Suppl 11):S22–S35.
14.	Noonan VK, Kopec JA, Noreau L, et al. Comparing the content of participation instruments using the International Classification of Functioning, Disability and Health. Health Qual Life Outcomes. 2009;7:93.
15.	Salter K, Jutai JW, Teasell R, Foley NC, Bitensky J, Bayley M. Issues for selection of outcome measures in stroke rehabilitation: ICF participation. Disabil Rehabil. 2005;27:507–528.
16.	Salter K, Jutai JW, Teasell R, Foley NC, Bitensky J, Bayley M. Issues for selection of outcome measures in stroke rehabilitation: ICF activity. Disabil Rehabil. 2005;27:315–340.
17.	Noonan VK, Miller WC, Noreau L; SCIRE Research Team. A review of instruments assessing participation in persons with spinal cord injury. Spinal Cord. 2009;47:435–446.
18.	Magasi S, Post MW. A comparative review of contemporary participation measures’ psychometric properties and content coverage. Arch Phys Med Rehabil. 2010;91(Suppl 9):S17–S28.
19.	Wilkie R, Jordan JL, Muller S, Nicholls E, Healey EL, van der Windt DA. Measures of social function and participation in musculoskeletal populations: Impact on Participation and Autonomy (IPA), Keele Assessment of Participation (KAP), Participation Measure for Post-Acute Care (PM-PAC), Participation Objective, Participation Subjective (POPS), Rating of Perceived Participation (ROPP), and The Participation Scale. Arthritis Care Res. 2011;63(Suppl 11):325–336.
20.	Tse T, Douglas J, Lentin P, Carey L. Measuring participation after stroke: a review of frequently used tools. Arch Phys Med Rehabil. 2013;94:177–192.
21.	Food and Drug Administration, Department of Health and Human Sciences. Guidance to Industry. Patient Reported Outcome Measures. Use in Medical Product Development to Support Labelling Claims. Silver Spring, MD, USA: Food and Drug Administration; 2009.
22.	Speight J, Barendse SM. FDA guidance on patient reported outcomes: a prompt for the industry to raise scientific standards. BMJ. 2010;340:c2921.
23.	Ware JE Jr, Sherbourne CD. The MOS 36-item short-form health survey (SF-36) I: Conceptual framework and item selection. Med Care. 1992;30:473–483.
24.	Ware JE Jr, Kosinski M, Bjorner JB, Turner-Bowker DM, Gandek B, Maruish ME. User’s Manual for the SF-36 Version 2. 2nd ed. Lincoln, RI, USA: Qualitymetric; 2007.
25.	Jenkinson C, Stewart-Brown S, Petersen S, Paice C. Assessment of the SF-36 version 2 in the United Kingdom. J Epidemiol Community Health. 1999;53:46–50.
26.	McDowell I. General health status and quality of life. In: McDowell I, editor. Measuring Health: A Guide to Rating Scales and Questionnaires. 3rd ed. Oxford, UK: Oxford University Press; 2006.
27.	Ware JE Jr, Kosinski M, Keller SD. SF-36 Physical and Mental Health Summary Scales: A User Manual. Boston, MA, USA: The Health Institute, New England Medical Center; 1994.
28.	Ware JE Jr, Kosinski M, Bayliss MS, McHorney CA, Rogers WH, Raczek A. Comparison of methods for the scoring and statistical analysis of SF-36 health profile and summary measures: summary of results from the Medical Outcomes Study. Med Care. 1995;33(Suppl 4):AS264–A279.
29.	EuroQol Group. EuroQol – A new facility for the measurement of health related quality of life. Health Policy. 1990;16:199–208.
30.	Hurst N, Kind P, Ruta D, Hunter M, Stubbings A. Measuring health-related quality of life in rheumatoid arthritis: validity, responsiveness and reliability of EuroQol (EQ-5D). Br J Rheumatol. 1997;36:551–559.
31.	Streiner DL, Norman GR. Devising the items. In: Streiner DL, Norman GR, editors. Health Measurement Scales: A Practical Guide to their Development and Use. 4th ed. Oxford, UK: Oxford University Press; 2008.
32.	Flesch R. A new readability yardstick. J Appl Psychol. 1948;32:221–233.
33.	Willis G. Cognitive Interviewing and Questionnaire Design: A Training Manual. Cognitive Methods Staff Working Paper Series No 7. Hyattsville, MD, USA: Office of Research and Methodology, National Center for Health Statistics; 1994.
34.	Norman GR, Streiner DL. Principal components and factor analysis. In: Norman GR, Streiner DL, editors. Biostatistics: The Bare Essentials. 3rd ed. Hamilton, Ontario, Canada: BC Decker Inc; 2008.
35.	Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol. 1997;50:1129–1136.
36.	Shapiro SS, Wilk MB. An analysis of variance test for normality (complete samples). Biometrika. 1965;52:591–611.
37.	Terwee CB, Bot SD, De Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60:34–42.
38.	Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. Chicago, IL, USA: University of Chicago Press; 1960.
39.	Andrich D. Rasch Models for Measurement. London, UK: Sage Publications; 1988.
40.	Hobart J, Cano S. Rasch analysis. In: Jenkinson C, Peters M, Bromberg M, editor. Quality of Life Measurement in Neurodegenerative and Related Conditions. Cambridge, UK: Cambridge University Press; 2011.
41.	Pearson K. Mathematical contributions to the theory of evolution. III. Regression, heredity and panmixia. Philos Trans R Soc Lond A. 1896;187:253–318.
42.	Spearman C. The proof and measurement of association between two things. Am J Psychol. 1904;15:72–101.
43.	Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.
44.	Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420–428.
45.	Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Mahwah, NJ, USA: Lawrence Erlbaum; 1988.