Back to Archived Journals » Comparative Effectiveness Research » Volume 5

Defining the benefits and challenges of stakeholder engagement in systematic reviews

Authors Cottrell E, Whitlock E, Kato E, Uhl S, Belinson S, Chang C, Hoomans T, Meltzer D, Noorani H, Robinson K, Motu'apuaka M, Anderson J, Paynter R, Guise J

Received 18 June 2014

Accepted for publication 19 August 2014

Published 16 April 2015 Volume 2015:5 Pages 13—19

DOI https://doi.org/10.2147/CER.S69605

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 3

Editor who approved publication: Dr Corrine I Voils



Erika K Cottrell,1 Evelyn P Whitlock,2 Elisabeth Kato,3 Stacey Uhl,4 Suzanne Belinson,5 Christine Chang,3 Ties Hoomans,5,6 David O Meltzer,5,7 Hussein Noorani,5 Karen A Robinson,8 Makalapua Motu'apuaka,9 Johanna Anderson,9 Robin A Paynter,9 Jeanne-Marie Guise9

1Oregon Health and Sciences University, Portland, OR, USA; 2Kaiser Evidence-based Practice Center, Portland, OR, USA; 3Agency for Healthcare Research and Quality, Rockville, MD, USA; 4ECRI-Penn Evidence-based Practice Center, Plymouth Meeting, PA, USA; 5Office of Clinical Affairs, Blue Cross Blue Shield Association, Chicago, IL, USA; 6Institute of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, the Netherlands; 7University of Chicago, Chicago, IL, USA; 8Johns Hopkins University Evidence-based Practice Center, Baltimore, MD, USA; 9Scientific Resource Center for the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program, Portland Veterans Affairs (VA) Research Foundation, Portland VA Medical Center, Portland, OR, USA

Background: Although there is a growing literature on the process of engaging stakeholders in medical research, there are a lack of clearly-defined measures for reporting and evaluation, which limits the ability to learn from past experience, understand the effectiveness of engagement, or identify which approaches work best. Clearly defining the benefits and challenges of engaging stakeholders in the systematic review process is an integral first step toward developing a set of criteria that can be used to evaluate the impact and effectiveness on the conduct, quality, and dissemination of systematic reviews.
Methods: We utilized two complementary approaches to examine the benefits and challenges of engaging stakeholders in the systematic review process: 1) a literature scan to understand the overall state of the field; and 2) a series of key informant interviews with systematic reviewers, program/policy officials, and stakeholders.
Results: We identified six main expected benefits and five primary challenges of involving stakeholders in systematic reviews. Benefits included: establishing credibility; anticipating controversy; ensuring transparency and accountability; improving relevance; enhancing quality; and increasing dissemination and uptake of findings. Challenges included: time; training and resources; finding the right people; balancing multiple inputs; and understanding how to match the right type of stakeholder to the right time in the systematic review process.
Discussion: The results of this study are an important first step toward developing mechanisms for evaluating the effectiveness of stakeholder engagement in systematic reviews. Future work should seek both to verify the appropriateness of these benefits and challenges and identify concrete criteria for evaluating the effectiveness of different methods, timing, and intensity of stakeholder engagement.

Keywords: stakeholder, medical research, key informants, AHRQ, Effective Health Care Program, evidence based medicine

Introduction

Background and rationale

Medical research has evolved over the years from the paradigm of the lone research scientist to a model that encompasses a broader range of clinical research activities, including multidisciplinary team science. One of the more recent developments in this evolution is the active engagement of stakeholders (ie, a person or group with a vested interest in a particular clinical decision and the evidence that supports that decision1) in the selection, design, funding, and conduct of medical research, including systematic reviews. According to the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care (EHC) Program, stakeholders include: patients, caregivers, and patient advocacy organizations; clinicians and their professional associations; institutional health care providers, such as hospital systems and medical clinics; government agencies; purchasers and payers, such as employers and public and private insurers; health care industry representatives; health care policymakers at the federal, state, and local levels; and health care researchers and research institutions.1 The overarching goal of stakeholder engagement in systematic reviews and other medical research is to generate evidence that is more relevant and useful to those making real-world health care decisions, with the hope that this will increase the dissemination and uptake of research findings in clinical practice.2,3 Activities to ensure engagement range from relatively simple approaches, such as asking stakeholders to participate in research meetings or comment on research proposals, to more resource-intensive methods, including the use of Delphi techniques to reach consensus about research priorities and involving stakeholders directly in making funding decisions or conducting research.4,5

Despite the growing emphasis on stakeholder engagement, there are a lack of clearly-defined measures for reporting and evaluation, which limits the ability to learn from past experience, understand the effectiveness of engagement, or identify what approaches work best.6 Clearly defining the benefits and challenges involved in this process is an integral first step toward developing criteria for evaluating the impact of stakeholder engagement on the conduct, quality, and dissemination of systematic reviews. Indeed, knowing how to make an activity “more effective” presupposes an understanding of the desired outcome or benefit as well as the challenges, tradeoffs, and adverse consequences. Toward this end, this paper examines the following questions: 1) what are the expected benefits of involving stakeholders in systematic reviews, and 2) what are the perceived challenges of involving stakeholders in systematic reviews?

We define expected benefits (hereafter referred to as benefits) as the expected positive changes to the quality and impact of the systematic review as a result of engaging stakeholders in the process. Additional positive externalities from engagement (ie, positive consequences of the process that did not directly affect the quality and impact of the review such as building relationships, educating stakeholders about the research process, and personal benefits of involvement) were not included in the scope of this report. We define perceived challenges (hereafter referred to as challenges) as the costs and difficulties associated with engagement that need to be balanced against the benefits of involvement.

Methods

To begin to answer these questions, we utilized two complementary approaches: 1) a literature scan; and 2) a series of key informant (KI) interviews.

Literature scan

We searched a range of databases to identify published and unpublished studies that address the benefits, challenges, and/or measurement of stakeholder engagement in order to: 1) understand the overall state of the field and whether there were existing efforts to explicitly define or evaluate the impact of stakeholder engagement in primary research or systematic reviews; and 2) better inform and focus the KI interviews. The complete search strategy is included in the full report by Cottrell et al.7

Three investigators reviewed 299 abstracts and 80 full text articles. We included articles on the benefits, challenges, and/or measurement of stakeholder engagement in systematic reviews or medical research more generally. We excluded articles that were published prior to 2005, those that primarily focused on how to engage stakeholders (instead of why or when), and those not relevant to medical research or systematic reviews. Twenty-four articles met the inclusion/exclusion criteria;4,6,826 only four articles were specific to systematic reviews.1416,18 Information regarding benefits, challenges, and/or measurement/evaluation of stakeholder engagement was extracted from these articles and synthesized in conjunction with the results of the KI interviews. See the full report by Cottrell et al for details on included studies.7

Key informant interviews

To supplement the information gleaned from the limited literature in this area, we conducted a series of semi-structured interviews with systematic reviewers who engage stakeholders in their work, program/policy officials who commission and/or use systematic reviews, and stakeholders who have been involved in the systematic review or research process.

A balance of representatives from each of the above groups was invited to take part in a 90-minute individual or group interview. Of the 60 potential KIs invited, 34 agreed to participate – 15 systematic reviewers, ten program/policy officials, and nine stakeholders. In all, we conducted 12 discussion sessions with between one and four participants each. Interviews were conducted by the lead author using a semi-structured interview guide designed to elicit a multi-faceted understanding of the expected benefits and perceived challenges of engaging stakeholders in systematic reviews. Although the primary purpose of the interviews was to understand the perceived benefits and challenges of stakeholder engagement as a necessary first step toward developing metrics and measures, we also asked interviewees for their suggestions on how such evaluation might be done. See the full report by Cottrell et al for the interview guide.7

Transcripts were entered into NVivo software for analysis. Two investigators and two research assistants read all of the transcripts to identify recurring themes, with an eye toward articulating the benefits and challenges of engaging specific types of stakeholders at each phase of the systematic review. Finally, we noted any particular suggestions by KIs of how to measure or evaluate the impact of engagement. Themes from the KI interviews and literature scan were reviewed with the larger working group during bi-weekly calls; disagreements were resolved through consensus.

Results

Overall, the themes from the KI interviews largely echoed the benefits and challenges outlined in the literature on stakeholder engagement in systematic reviews and medical research more generally. However, KIs provided a more nuanced understanding of the tradeoffs involved and augmented the nascent literature with specific insights into the benefits and challenges of engaging stakeholders in systematic reviews.

In short, expected benefits identified by KIs and in the literature clustered into six overarching domains: 1) establishing credibility;6,815 2) anticipating controversy;10 3) ensuring transparency and accountability;6,810,1320 4) improving relevance;6,810,1222 5) enhancing quality;6,810,1224 and 6) increasing dissemination and uptake of systematic review findings.6,9,10,1215,1822 Perceived challenges included: 1) time;10,12,14,17,18,20,22,23,25,26 2) training and resources (both for stakeholders as well as systematic reviewers);4,10,12,14,17,18,2123,25 3) finding the right people;9,10,14,17,22,23,25 4) balancing multiple, often competing inputs;10,11,14,23 and 5) understanding when to engage specific types of stakeholders.10,14,23 We discuss each of the overarching benefits and challenges below using context from the KI interviews. Additional details on the results, including descriptions of the articles reviewed in the literature scan and data from the KI interviews, are available in the full report by Cottrell et al.7

What are the expected benefits of involving stakeholders in systematic reviews?

Establishing credibility

One of the immediate benefits is establishing credibility and gaining buy-in from the stakeholder community.6,815 Systematic reviewers and program/policy officials in particular suggested that engaging stakeholders from the beginning increases the likelihood that they truly understand the process and feel that their insights are incorporated into the design and conduct of the review. KIs who had been engaged in a review as a stakeholder reported that it gave them a sense of investment in the process and findings, as opposed to feeling that the results were simply “announced” to them at the end.

Anticipating controversy

Several KIs explained that a key benefit of engaging stakeholders is the ability to anticipate controversies and opposing views that could potentially derail a report’s relevance. Systematic reviewers and program/policy officials in particular underscored the imperative of understanding the context and history of a given topic, including areas of scientific uncertainty or debate and politically charged or “hot button” issues from a consumer or advocacy perspective. Moreover, they described instances when stakeholders were able to uncover “blind spots” including issues of current debate or concern that impacted what questions were asked or how the report’s findings were communicated.10

Ensuring transparency and accountability

Engaging stakeholders in the review process was described as a key aspect of ensuring transparency and accountability, particularly in situations where systematic reviews directly determine policy.6,810,1320 KIs from the stakeholder and program/policy perspective suggested that engaging a broader community in the process is a democratic right or moral imperative. Systematic reviewers and program officials both noted that mandates to engage stakeholders are important aspects of displaying accountability, transparency, and ongoing support for systematic reviews.

Improving relevance

Another primary motivation for engaging stakeholders is the belief that engaging the end users – patients, clinicians, policymakers, or guideline developers – is necessary to produce reports that are timely and relevant.6,810,1222 There was general agreement that engaging those who would use the evidence was the best way to ensure that a systematic review addresses the right questions, includes the right outcomes, and that the review team does not miss a critical perspective. KIs repeatedly emphasized that stakeholders bring an understanding of the context that comes from having in-depth professional or personal experience with a topic; without this input, systematic reviews could miss the target and end up gathering dust on shelves rather than changing practice.

Enhancing quality

Perhaps one of the strongest overriding themes was the sense that engaging stakeholders would improve the overall quality of the review.6,810,1224 Systematic reviewers and program officials identified a number of specific areas where stakeholder input improved the scientific quality of the final report, including: 1) framing the review and defining the key questions; 2) helping the team to refine the scope of the review; and 3) establishing appropriate parameters for the population, intervention, comparator, outcome, timing, and settings considered. Systematic reviewers credited stakeholder input with identifying nuances that were integral in helping to clearly define the population, intervention, comparator, outcome, timing, and settings considered in a given review. Moreover, clinical experts were identified as instrumental in providing valuable context that may not be readily apparent from the published literature, including identifying tests and procedures that are obsolete, or highlighting dosing and side effect profiles of different classes of medications that may influence study findings.

Increasing dissemination and uptake of systematic review findings

All of our discussions and several of the articles reviewed touched in some fashion on the idea of engaging stakeholders as a route to improving the dissemination and uptake of systematic review findings.6,9,10,1215,1822 KIs from all perspectives agreed with the principle that involving stakeholders meaningfully in the process should, at least in theory, increase the likelihood that systematic reviews address the right questions; are relevant to the real-life decisions faced by patients, clinicians, and policymakers; and that the findings are actually used to guide and inform real-life medical decisions. Moreover, KIs expressed the hope that by involving stakeholders in the conduct of systematic reviews, they were more likely to become advocates for the results, thereby facilitating more widespread dissemination and uptake of the final product.

What are the perceived challenges of involving stakeholders in systematic reviews?

Time

KIs from all perspectives agreed that engaging stakeholders requires a significant investment of time, especially if it is to be done well; this idea resonated in the literature as well.10,12,14,17,18,20,22,23,25,26 Systematic reviewers repeatedly emphasized that engaging stakeholders in the systematic review process lengthens the project timeline, especially if stakeholders are engaged at more than one time point. Given how rapidly new research becomes available, this can be a significant problem for systematic reviews. There was disagreement as to whether the extra time involved resulted in an improved product, with some arguing that the time spent engaging stakeholders was at least partly recouped by avoiding missteps, and others arguing that getting the results out to stakeholders earlier was more important than engaging them in the systematic review process. This investment of time was also cited as an issue for stakeholders, who usually have other professional or caregiving responsibilities.

Training and resources

In addition to the extra time involved, a lack of appropriate training and resources can also limit the benefits of stakeholder engagement.4,10,12,14,17,18,2123,25 Many investigators are unfamiliar with how to best utilize and engage stakeholders and lack the skills required for successful management of such a process. Moreover, stakeholders who do not have a clinical or research background may require additional training and ongoing support in order to make a meaningful contribution to the process. KIs from all perspectives, but especially systematic reviewers and those who had served as stakeholders, suggested that ensuring that both sides have the necessary background and training, and sufficient resources to support their role, would go a long way toward increasing the overall benefit of stakeholder engagement in systematic reviews.4,10,12,14,17,18,2123,25

Finding the right people

Identifying and inviting stakeholders to participate in the review process is not an exact science and figuring out who to engage can be a challenge.9,10,14,17,22,23,25 Moreover, because US Office and Management and Budget Paperwork Reduction Act regulations (the Office of Management and Budget Regulations, USA) limit the number of stakeholders who can be engaged in a given review conducted for the federal government, the choice of who to engage carries a special weight in those cases. Getting it wrong can limit or negate the value of the engagement. During one discussion among systematic reviewers, there was agreement that in many instances the benefit of engagement is highly correlated with which types of stakeholders are engaged and/or the make-up of the stakeholder group.

KIs who had been involved as stakeholders and systematic reviewers alike noted that not all stakeholders have the same level of commitment and/or ability to contribute significantly to the systematic review process.9,10,14,17,22,23,25 Engaging those with high commitment runs the risk of engaging those with the most conflicts of interest. Moreover, as highlighted above, stakeholders who are not intimately familiar with the scientific review process may need substantial training in order to make valuable contributions.

Both stakeholders and systematic reviewers also emphasized that beyond their clinical expertise or personal experience with the topic, less tangible individual characteristics often play a large role in shaping the value and impact of stakeholder engagement, including personality traits, level of social influence among peers, level of personal motivation and willingness to engage, comfort with sharing opinions among a room full of “experts”, and comfort with group processes. Some KIs even suggested developing a database of “good stakeholder” contacts so that those who are a proven asset to this type of work can be accessed for future work.

Balancing multiple inputs

Another challenge is defining the relative value of different stakeholder input.10,11,14,23 Only a limited number of stakeholders are engaged per review and it can be a challenge to determine whether a particular stakeholder is an outlier or has a personal or political agenda. As one systematic reviewer emphasized, not all input is equal and it is often difficult to assess which points to incorporate. It is a delicate balance to make stakeholders feel that their insights are valued, while at the same time being mindful that not all ideas can be incorporated into the review given both scientific and resource constraints. Some systematic reviewers expressed concern about expectations associated with being able to respond to all opinions expressed through stakeholder engagement. On the other hand, KIs who had served as stakeholders reported feeling pressure to come up with new or interesting insights and contributions or raise issues that the systematic review team had not addressed. Stakeholders also reported feeling intimidated by the process or under pressure to serve as the representative voice for a large population of people.

Understanding how to match the right type of stakeholder to the right time in the systematic review process

An overarching theme from both systematic reviewers and stakeholders who had been engaged in a systematic review was the concern that mandates for more extensive stakeholder engagement in systematic reviews seem to suggest that “more is better”, but there has been less attention to understanding when in the process it is most useful to engage particular types of stakeholders.10,14,23 There is a danger of “lumping” all stakeholders together into a homogenous group, without recognizing the relative value of different contributions at various points in the review process.

Discussion

Although the findings of our KI interviews largely echo the benefits and challenges outlined in literature,4,6,829 KI interviews provided a more nuanced understanding of tradeoffs, including the variation in relative benefits and challenges depending on who is engaged and when they are engaged. Systematic reviewers in particular highlighted the importance of engaging the right stakeholders at the right time and explained that the benefit of engagement often depends on the characteristics of the stakeholders involved. They also noted the challenge of deciding what input to incorporate into the report, and the inability to respond to all stakeholder comments and/or concerns.

Overall, one of the strongest messages from the KI interviews was the importance of linking the benefits of engaging specific types of stakeholders at different stages of the review process, rather than simply engaging a broad range of stakeholders in all phases of the review as “insurance” or in response to programmatic requirements. Of all phases of the systematic review, the “Topic refinement and research development” phase was repeatedly identified as the point where stakeholder engagement yielded the greatest benefit, with a preponderance of comments focused on the importance of stakeholder engagement at this phase in order to get buy-in and credibility for the process and product, to ensure the systematic review was scientifically valid, relevant, and useful, and to ensure uptake and use of the review products.

Despite a general agreement on the utility, neither the literature scan nor the KI interviews revealed any explicit efforts to evaluate the effectiveness or impact of stakeholder engagement. When we asked KIs to suggest possible methods, the most frequently cited was to evaluate at the impact of stakeholder engagement by assessing the quality, relevance, and dissemination and uptake of a review. For example, KIs suggested that tracking how a review is used to guide policy decisions or to inform guideline development may be an important indicator of the impact of engagement. A number of systematic reviewers mentioned the possibility of looking at this question retrospectively by comparing the quality and usefulness of past reports with varying degrees of engagement. Other KIs proposed pragmatic measures of the effectiveness, such as tracking the changes in key questions, inclusion and exclusion criteria, or overall scope of the review.

The results of this study are an important first step toward developing mechanisms for evaluating the effectiveness of stakeholder engagement in systematic reviews. Although the findings of our KI interviews are limited by a relatively small sample, the themes are similar to those found in the handful of papers on stakeholder engagement in systematic reviews1416,18 and the relatively more developed (yet still nascent) literature on stakeholder engagement in medical research more generally.4,6,813,17,1926 The congruence between themes suggests that the benefits and challenges of stakeholder engagement in systematic reviews may be generalizable to medical research, and vice versa. Future work should seek both to verify the appropriateness of the set of benefits and challenges highlighted in this paper and to take the next step of identifying criteria for evaluating the effectiveness of different methods, timing, and intensity of stakeholder engagement.

Acknowledgments

This project was funded under Contract 290-2012-00004-C from the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services.

Author contributions

EKC conducted all interviews, participated in conception and design of the project, analyzed and interpreted project material, and drafted the manuscript. EPW helped analyze and interpret project material and critically revised the manuscript. EK helped analyze and interpret project material and critically revised manuscript. SU helped analyze and interpret project material and critically revised the manuscript. SB helped analyze and interpret project material and critically revised the manuscript. CC helped analyze and interpret project material and critically revised the manuscript. TH helped analyze and interpret project material and critically revised the manuscript. DOM helped analyze and interpret project material and critically revised the manuscript. HN helped analyze and interpret project material and critically revised the manuscript. KAR helped analyze and interpret project material and critically revised the manuscript. MM helped analyze project material and helped to draft the manuscript. JA helped analyze project material and helped to draft the manuscript. RAP helped analyze and interpret project material and critically revised the manuscript. JMG participated in conception and design of the project and helped draft the manuscript. All authors read and approved the final manuscript.

Disclosure

The author reports no conflicts of interest in this work. This paper is based on a Research White Paper produced for the Agency for Healthcare Research and Quality (AHRQ), and was posted to the AHRQ EHC Program Web site on March 18th 2014. The authors of this document are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.


References

1.

Effective Health Care Program: Stakeholder Guide 2014. Agency for Healthcare Research and Quality; [updated February 2014]. Available from: http://www.ahrq.gov/research/findings/evidence-based-reports/stakeholderguide/stakeholdr.pdf. Accessed March 1, 2013.

2.

Methodology Committee of the Patient-Centered Outcomes Research Institute (PCORI). Methodological standards and patient-centeredness in comparative effectiveness research: The PCORI perspective. JAMA. 2012;307(15):1636–1640.

3.

Initial National Priorities for Comparative Effectiveness Research. Institute of Medicine of the National Academies; 2009. Available from: http://www.iom.edu/~/media/Files/Report%20Files/2009/ComparativeEffectivenessResearchPriorities/CER%20report%20brief%2008-13-09.ashx. Accessed March 1, 2013.

4.

Curtis P, Slaughter-Mason S, Thielke A, et al. PCORI Expert Interviews Project: Final Report. Center for Evidence-based Policy, Oregon Health and Science University; 2012. Available from: http://pcori.org/assets/pdfs/Expert%20Interviews%20Part%201.pdf. Accessed March 1, 2013.

5.

Guise J-M, O’Haire C, McPheeters M, et al. A practice-based tool for engaging stakeholders in future research: a synthesis of current practices. Journal of clinical epidemiology. 2013;66(6):666–674.

6.

Brett J, Staniszewska S, Mockford C, Seers K, Herron-Marx S, Bayliss H. The PIRICOM Study: A Systematic Review of the Conceptualisation, Measurement, Impact and Outcomes of Patients and Public Involvement in Health and Social Care Research. Royal College of Nursing; The University of Warwick; 2010. Available from: http://www.ukcrc.org/wp-content/uploads/2014/03/Piricom+Review+Final+2010.pdf. Accessed March 1, 2013.

7.

Cottrell E, Whitlock E, Kato E, et al. Defining the Benefits of Stakeholder Engagement in Systematic Reviews. Research White Paper. [Prepared by the Scientific Resource Center under Contract No 290-2012-00004-C]; Agency for Healthcare Research and Quality; 2014. Available from: http://effectivehealthcare.ahrq.gov/ehc/products/581/1883/stakeholder-engagement-benefits-report-140318.pdf. Accessed March 1, 2014.

8.

Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database of Systematic Reviews. Issue 3: CD004563 2006. Available from: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD004563.pub2/abstract. Accessed March 1, 2013.

9.

Brett J, Staniszewska S, Mockford C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health expectations. 2012;17(5):637–650.

10.

Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in systematic reviews: Knowledge transfer for policy and practice. Journal of Continuing Education in the Health Professions. 2008; 28(2):67–72.

11.

Hoffman A, Montgomery R, Aubry W, Tunis SR. How best to engage patients, doctors, and other stakeholders in designing comparative effectiveness studies. Health Affairs. 2010;29(10):1834–1841.

12.

Domecq Garces JP, Prutsky Lopez GJ, Wang Z, et al. Eliciting Patient Perspective in Patient-Centered Outcomes Research: A Meta Narrative Systematic Review. Patient-Centered Outcomes Research Institute; 2012. Available from: http://www.pcori.org/assets/Eliciting-Patient-Perspective-in-Patient-Centered-Outcomes-Research-A-Meta-Narrative-Systematic-Review.pdf. Accessed March 1, 2013.

13.

Kreis J, Puhan MA, Schunemann HJ, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expectations. 2012;16(4):323–337.

14.

Public Involvement in Systematic Reviews: Supplement to the Briefing Notes for Researchers. INVOLVE; National Institute for Health Research; National Health Service; 2012. Available from: http://www.invo.org.uk/wp-content/uploads/2012/10/INVOLVEPublicInvolvementSystematicReviews2012.pdf. Accessed March 1, 2013.

15.

Wright D, Foster C, Amir Z, Elliott J, Wilson R. Critical appraisal guidelines for assessing the quality and impact of user involvement in research. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy. 2010;13(4):359–368.

16.

Boote J, Barber R, Cooper C. Principles and indicators of successful consumer involvement in NHS research: results of a Delphi study and subgroup analysis. Health policy (Amsterdam, Netherlands). 2006;75(3):280–297.

17.

Boote J, Baird W, Sutton A. Involving the public in systematic reviews: a narrative review of organisational approaches and eight case examples. Journal of Comparative Effectiveness Research. 2012;1(5):409–420.

18.

Vale CL, Thompson LC, Murphy C, Forcat S, Hanley B. Involvement of consumers in studies run by the Medical Research Council Clinical Trials Unit: results of a survey. Trials. 2012;13(9):1–10.

19.

Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J. Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach (Structured abstract). Health Technology Assessment Database. 2004;8(15):1–148.

20.

Barber R, Boote JD, Parry GD, Cooper CL, Yeeles P, Cook S. Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy. 2012;15(3):229–241.

21.

Deverka PA, Lavallee DC, Desai PJ, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. Journal of Comparative Effectiveness Research. 2012;1(2):181–194.

22.

Hailey D, Werko S, Bakri R, et al. Involvement of Consumers in Health Technology Assessment Activities by Inahta Agencies. International Journal of Technology Assessment in Health Care. 2012;29(1):79–83.

23.

Horey D. Consumer Involvement in the Cochrane Collaboration. Background Paper. 2010. Available from: http://consumers.cochrane.org/sites/consumers.cochrane.org/files/uploads/Consumer%20Involvement%20in%20the%20Cochrane%20Collaboration%20Background%20paper.pdf. Accessed March 1, 2013.

24.

Wale J, Colombo C, Belizan M, Nadel J. International health consumers in the Cochrane Collaboration: fifteen years on. Journal of Ambulatory Care Management. 2010;33(3):182–189.

25.

Hailey D. Consumer Involvement in Health Technology Assessment. Alberta Heritage Foundation for Medical Research; 2005. Available from: http://www.ihe.ca/documents/HTA-FR21.pdf. Accessed March 1, 2013.

26.

Gagnon M-P, Desmartis M, Lepage-Savary Ds, et al. Introducing patients’ and the public’s perspectives to health technology assessment: A systematic review of international experiences. International Journal of Technology Assessment in Health Care. 2011;27(1):31–42.

Creative Commons License © 2015 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.