Back to Journals » Patient Related Outcome Measures » Volume 13

A Systematic Review of Patient Engagement Experiences in Brain Disorders

Authors Bertorello D, Brichetto G, Folkvord F, Theben A, Zaratin P

Received 7 June 2022

Accepted for publication 12 November 2022

Published 13 December 2022 Volume 2022:13 Pages 259—272


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Robert Howland

Deborah Bertorello,1 Giampaolo Brichetto,1 Frans Folkvord,2,3 Alexandra Theben,4 Paola Zaratin1

1Italian Multiple Sclerosis Society Foundation, Genoa, Italy; 2Open Evidence Research, Barcelona, Spain; 3Department of Communication and Cognition, Tilburg School of Humanities and Digital Sciences, Tilburg University, Tilburg, the Netherlands; 4Department of Management, Humanities, Law and Society, UPF-Barcelona School of Management, Barcelona, Spain

Correspondence: Deborah Bertorello, Italian Multiple Sclerosis Society Foundation, Via Operai, 40, Genoa, 16149, Italy, Tel + 39 010 2713 247, Fax + 39 010 2713 205, Email [email protected]

Background: Patient engagement is increasingly considered to be an important element in the treatment of brain disorders to optimise outcomes for patients, society, and healthcare systems. Nonetheless, scientific research examining methodologies to engage patients with brain diseases in Research and Innovation (R&I) is scarce.
Aim: To review existing scientific evidence regarding the engagement of patients with brain disorders in research and innovation.
Methods: Studies were retrieved from several bibliographic databases (publication date between January 2016 and April 2019) with pre-specified selection criteria.
Results: In total, 49 articles were identified as meeting the inclusion criteria and were reviewed systematically. Results showed that there is limited evidence available on the impact and (cost-) effectiveness of patient engagement in (brain) research and innovation. Most published studies are protocols, guidelines, and discussion articles for patient engagement in health research and innovation. Overall, there exists a general consensus to engage patients in every step of the research procedure. Relevant evidence identified includes principles of engagement, definitions of stakeholder types, key considerations for planning, conducting and disseminating engaged research, potential engagement activities, and examples of promising practices.
Discussion: Findings are inconclusive due to methodological differences. Comparison between studies was difficult due to differences in patients, form of engagements, and total duration of engagement of patients. Experiences of patient engagement mainly concern adherence to medical treatments or participation of “expert patients” in clinical trials, but very rarely the governance of R&I according to the dictates of Responsible Research and Innovation (RRI). More structuralized, well-conducted and comparable Randomized Controlled Trials (RCTs) are needed to be able to make evidence-based recommendations on how to increase effective patient engagement in research and innovation and assess the impact and (cost)-effectiveness.

Keywords: systematic review, patient engagement, responsible research and innovation, brain disorders, patient reported outcomes


Patient engagement can be described as actions that patients must conduct to obtain the greatest benefit from the healthcare services available to them.1 Scholars worldwide agree on the urgency of engaging patients in their care in order to achieve a more sustainable management of the healthcare system2,3 also in mental health research.4,5 To make it more targeted and effectively, Graffigna et al6 have promoted and disseminated an Italian Consensus Conference on Patient Engagement (ICCPE), in order to set the basis for drafting recommendations for the provision of effective patient engagement interventions and research. Reliance on the patient’s knowledge, skills and motivation to access the acquired benefits through the advances in medicine, technology and healthcare services are increasingly necessary in order to improve the outcomes of the health interventions.6

As Graffigna et al6 formulated, this contributes to a wide “system” inertia - one that is really difficult to be overcome - and puts the research field at risk for any forms of innovation, although studies have come up with a framework for advancing the reporting of patient engagement in research projects.7 In general, the level of patient engagement is largely influenced by institutional ideologies, professional attitudes and the readiness of patients to accept new and engaging roles.8 The level of patient engagement is also influenced by the status and acceptance of the disease; Graffigna et al9 have developed a model to understand this level of acceptance. Within this context, patient engagement is vital in the treatment of brain diseases in order to optimise health outcomes for patients, society, and healthcare systems.10 Current advances in medicine, technology and healthcare services offer promises of longevity and improved quality of life.11–14 More specifically, advances in digital technology has the potential to transform mental healthcare by connecting patients, services and health data in new and efficient ways that could lead to more tailored and personalized health interventions. As a consequence, these interventions will be more (cost-) effective than traditional forms of health interventions.5,12,15,16 For example, digital and mobile applications can offer patients greater access to information and services and enhance clinical management and early intervention through access and usage of real-time patient data.12

In addition, recent studies have shown that key characteristics of big data and how medical and health informatics, translational bioinformatics, sensor informatics, and imaging informatics can expand our knowledge to test for new hypotheses about brain diseases management, from diagnosis to prevention to personalized treatment.12,17 However, substantial gaps still exist in the evidence base underlying the adoption and usage of these new technologies. Zafra-Tanaka et al18 recently showed that clinical practice guidelines should follow adequate methodologies using an evidence-bases approach to provide reliable and valid recommendations, but most evaluated clinical practice guidelines did not take into account the patient’s viewpoints and did not clearly formulate the process used to reach the recommendations. The same applies for the development of Patient Reported Outcomes (PRO) that have not been developed by taking into consideration the patient “voice”.19,20

In order to make effective use of new health-related technological developments, it is essential to involve and engage all stakeholders continuously in developing and testing potential solutions and clinical guidelines, and working in a multidisciplinary way with all stakeholders to ensure that people with brain diseases are included in shared decision-making and disease management.21,22 Moreover, patient engagement has been labelled as the ‘blockbuster drug of the century’23 for upcoming and rapidly developing methodologies like health information technology24 and personal health records.17,25 Rationales for patient engagement in developing clinical guidelines include recognizing patients as expert in their experience and perception of the disease, with important contributions, thereby empowering them in well-informed healthcare decisions and respecting the rights of citizens in healthcare policy. Current goals and recommendations for brain research are to follow more patient-centred, trustworthy and effective guidelines in future research that lead to improved implementation and quality of care.26

As a first step to overcome this wide “system” inertia in brain research and innovation (R&I), an overview of the existing literature is needed. Therefore, we conducted a systematic review to examine patient engagement in R&I, with the aim of making evidence-based recommendations about the use of patient engagement in future research in brain diseases.


For the current systematic review, the following bibliographic databases were searched: PubMed, Cochrane Library, PsychInfo and EMBASE. The bibliographic search was conducted in March 2019 and was restricted to peer-reviewed papers or dissertations written in English between 2016 and 2019, because this was part of a larger project (MULTI-ACT,,73 that focused on studies conducted in this timeframe. Additional papers were identified by manually searching the reference lists of the retrieved articles and previous systematic reviews.

The search strategy that was used was conducted with three different levels. At the first level, the following search words were used:

Patient engagement, patient participation, patient involvement, patient support, patient co-creation, patient empowerment, patient consultation, patient decision making.

At level 2, the following search words were used: “research, research and development, research and innovation”. At level 3, the following search words were used: “guideline, protocol, practice guideline, recommendations, practices, and best practices”.

Two independent reviewers (FF and AT) conducted the search and screened the search results looking for studies that were considered eligible according to the information provided in the abstracts. Disagreements between reviewers were resolved by consensus.

Inclusion Criteria

The inclusion criteria were studies that reported on patient engagement or participation in research and innovation or responsible research. Furthermore, we were specially looking for studies reporting patients with brain diseases. For the search strings used, no studies were published focusing only on patients with brain diseases or related to brain research specificly, therefore we included also studies reporting patient engagement in health in general in order to be able to compare different studies with each other and make comparisons between different disease areas. Furthermore, we conducted forward and backward searching strategies by reading the references in the included studies in order to be sure we did not miss any relevant articles. These articles were included by additional manual search. In total, 49 articles that met the inclusion criteria were included in the narrative synthesis (see Figure 1).

Figure 1 PRISMA Flow diagram of selection of papers.

Notes: PRISMA figure adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of clinical epidemiology. 2009;62(10). Creative Commons.27


We categorized the results of the literature review based on the main focus of the studies. First, we will discuss the methods and steps of engagement that have been implemented and conducted by researchers in order to better understand how patient engagement exactly is established. Second, we will describe the best practices, barriers and recommendations of patient engagement as lessons learned. Third, we will show the outcomes of studies on the impact and return on investment of patient engagement in R&I. Finally, we will show the metrics that have been developed to evaluate the impact/return on investment of patient engagement in R&I.

Methods and Steps of Engagement

Engaging patients, caregivers, and other healthcare stakeholders as associates in every step of the research process has been discussed widely in the literature as key to effective patient engagement. Including all stakeholders in planning, conducting, and disseminating research is found to be a promising way to improve clinical decision-making and outcomes.28 Many researchers, patients, and stakeholders, however, lack clarity about when and how to engage as partners within the clinical research process. Most studies that were found discussed their experiences of patient engagement and developed a framework for future research on how to involve patients in research and innovation.

One of them is the Patient-Centred Outcomes Research Institute Engagement Rubric (Rubric) developed by the Patient-centred Outcomes Research Institute (PCORI).28 The Rubric provides a framework for operationalising the integration of patient engagement and other stakeholders in all phases of research. Importantly, it includes principles of engagement, definitions of stakeholder types, key considerations for planning, conducting and disseminating engaged research, potential engagement activities, and examples of promising practices from PCORI-funded projects. For example, there was a study conducted for stroke survivors, whereby the participants identified the number of days living at home and not in an institution or the hospital as an important outcome to measure. In addition, the neurology study, whereby the research team (including patient partners) presented information about the study at a neurology patient advocacy conference and alerted those attending about when to expect the results of the study. Wilson et al29 showed that it is very important to include patients in the selection and development of outcome assessments for medical product development, and therefore developed a framework for future studies.29

More specific and related to brain research, Rae-grant et al30 discussed in their practice guideline recommendations that patients should be involved more in the research and innovation for disease-modifying therapies for adults with multiple sclerosis (MS). The MS in the 21st Century Steering Group has been set out to foster patient engagement through a series of open-forum joint workshops.31 In order to increase patient engagement in research and innovation, they conducted two workshops with a diverse pool of important stakeholders in MS care, including neurologists, an MS nurse, a health economic specialist, a patient group representative, a neuro-rehabilitation specialist and a neuropsychologist.31 These workshops have shown that seven overall principles should support research and treatment of MS.

These principles include personalized care, patient engagement, commitment to research, regulatory body education and reimbursement issues, new endpoints in clinical trials, more therapy options, and MS centres of excellence. The action with regard to patient engagement in MS is devised on a set of themes.31 These cover: 1) setting and facilitating engagement education and confidence-building; 2) increasing the importance placed on quality of life and patient concerns through patient-reported outcomes; 3) providing credible sources of accurate information; 4) encouraging treatment adherence through engagement; and 5) empowering through the provision of sense of responsibility.

Köpke et al32 showed the importance of including MS-patients in the formulation of guideline questions, using mixed-methods (international online survey in eight countries, after pilot-testing debriefing on patients, and organizing focus group meetings among MS patients and their caregivers). The involvement was resource -and time- intensive, but at the same time very rewarding because it was the key for the formulation of 10 guidelines questions and for the identification of patient-relevant outcomes. Morby et al33 consulted people living with dementia and care partners to design an accessible Delphi survey to facilitate participation in core outcomes set development. The authors performed several steps. Firstly, they conducted face-to-face sessions to facilitate the development of a scale, and subsequently translated the 54 outcome areas into “accessible statements” for a two-round Delphi survey administered to five stakeholder groups (people living with dementia, care partners, health and social care professionals, policy-makers, and researchers). Finally, these steps led to the eventual delivery of a Delphi survey.

In addition, Murtagh et al34 showed in their ECOUTER-methodology for stakeholder engagement in translational research that characteristics such as flexibility, adaptability and openness are important elements for successful stakeholder engagement. ECOUTER uses mind-mapping techniques to open up engagement, both iteratively and organically. It aims to balance the breadth, accessibility and user-determination of the scope of engagement. The ECOUTER-methodology comprises four different stages; (1) engagement and knowledge exchange, (2) analysis of mindmap contributions, (3) development of a conceptual schema, and (4) feedback, refinement and development of recommendations.

Jennings et al35 conducted a critical literature review on patient and public involvement (PPI) in research to develop a methodology for involving PPI researchers in collaboratively analysing qualitative mental health research data with academic researchers. After piloting and refining the methodology, a best practice framework for collaborative data analysis of qualitative mental health research was created on the basis of the evidence gathered on successful involvement. The authors have shown that four collaborative data analysis approaches can be identified, namely: (1) consultation, (2) development, (3) application and (4) development and application of a coding framework. The collaborative data analysis is co-produced, realistic regarding time and resources, and demands of the process are manageable for patient and public involved researchers. In addition, group expectations and dynamics are effectively managed. This study shows the importance of developing a typology of approaches to collaboratively analysis of qualitative data in mental health research, and of identifying from available evidence the characteristics of successful involvement.

Adams et al36 developed the “Steps Model”, which is a practical tool for engaging communities to improve health-related outcomes and uses different steps to show that all parties must remain sensitive to one another’s needs. The model further emphasises the importance of maintaining the willingness to go down steps along the way and to rebuild the partnership if necessary. The tool shows that in order to build trust over time, it is necessary to develop and communicate about mutually beneficial outcomes and construct clear metrics for assessing impact.

Armstrong et al21 propose a ten-step framework outlining steps and options to increase patient engagement in clinical practice guideline development. At the developer level, patients can assist in topic nomination (step 1), topic prioritization (step 2), and guideline development group selection (step 3). Within the specific guideline projects, patients’ options may be better incorporated when framing the question (step 4), creating an analytic framework and research plan (step 5), conducting the systematic review and conclusion formation (step 6), development of recommendations (step 7), and dissemination and implementation (step 8). At the end of the process, patients can once more be engaged at the developer level by helping determine when guidelines need an update (step 9) and evaluate the developer’s approach to patient engagement (step 10).

Best Practices, Barriers and Recommendations

Clorafi24 showed that there are numerous initiatives underway to use health information technology to support patient engagement. However, the use of health information technology and other factors such as health literacy may be significant barriers to actually engage older adults. Methods to motivate patient engagement in research and innovation as reported in some of the studies include financial incentives. In particular, millennials consider themselves to be immune to poor health and underestimate the potential of developing health problems, such as brain diseases, and therefore less likely to adopt and use personal health records technologies.37 Arauwou38 showed that modifying older adult’s perceptions to use a patient portal for engagement in their healthcare is important.

Strategies include the provision of adequate training to help them explore the capabilities of a patient portal in monitoring health; receiving support of caregivers at healthcare facilities to use their influence to interact and help older adults to navigate through a patient portal; and receiving caregivers´ and physicians´ support to develop frequent correspondence with older adults through the patient portal. In similar strands, Gabel et al39 show that glioma patients should be involved in developing health-related quality of life outcomes to improve the metrics for future use in larger clinical research and clinical trial settings. Robillard et al40 in turn claim that patient engagement and research ethics collide and that bridging the gap between researchers and patients calls for reforms of current standards in dementia research.

Furthermore, Blackwell et al41 have shown that experience-based co-design is a useful approach for encouraging collaborative working between vulnerable patients, family and staff in complex healthcare environments, as in the case of patients with brain diseases. Grant et al42 analysed the practical considerations for using online methods to engage patients in the development of guidelines and found that online methods can facilitate greater openness and honesty by patients. They further support an adequate reflection of the diversity of patients´ views, which in turn improves the utility of clinical practice guidelines. Challenges include the fact that using online methods require extra skills, time and certain types of resources that may be needed for patient engagement. Ghisoni et al4 held a one-day workshop named “Getting involved in research: priority setting” to establish ideas and suggestions for research priorities from people who have experience of mental health services and found it was an efficient way to involve patients to a larger extent.

Nguyen et al43 argue that it is important to engage youth and families in research, in all the steps that need to be conducted. In addition, patients should be (1) involved in all aspects relate to designing the proposals, including the development of meaningful questions, (2) involved in co-production of the process to be used during and throughout the project, (3) named as investigators and members of the leadership team, (4) engaged in the analysis and interpretation of findings, (5) engaged in the dissemination of findings and results through reports, articles, presentation, and potentially as co-authors. When working with people with disabilities, it is important to provide clarity about roles, power, and authority to ensure all member’s contributions are equally valued while expectations are managed adequately. In addition, an environment of co-learning, trust, respect, reciprocity and shared decision-making should be created.43

Baines and de Bere44 have assessed the active involvement of patients and the public through an extensive systematic review and identified nine principles covering areas such as health and social care services, research, education and regulation across medicine, dentistry and nursing. They found that (1) working in equal partnership and (2) sharing information achieved the highest consensus rate by experts that reviewed the literature. This was followed by (3) communication and information provision (4) listening, assessing and responding (5) supporting and preparation, and (6) acknowledgement, reward and value for everyone involved. Lastly, it involves the (7) accommodation of individual and collective needs, (8) evaluation and (9) a tailored working-together approach as important principles that should be taken into account when considering patient and public engagement in health and social care services, research, education and regulation across medicine, dentistry and nursing.

Also, Simblett et al45 conducted a systematic review and show that there are different barriers and facilitators to engage patients with remote measurement technologies to manage their health. The review reveals that health status, perceived utility and value, motivation, convenience and accessibility, and usability are among the most commonly mentioned factors that explain usage of remote measurement technologies.

Rashid et al46 discussed that improving the recruitment of guideline group chairs, widening evidence reviews to include patient preference studies, adapting guidance presentation to highlight patient preferences points and providing clearer instructions on how patient organisation can submit their intelligence in research and innovations are emerging proposals that may help overcome barriers experienced by patients to further enhance patient and public involvement in their processes. One example is the study protocol by Samalin et al47 that will examine the efficacy of shared decision-making on treatment adherence of patients with bipolar disorder. To engage the community in research, Sand et al48 propose a “dyad” model, whereby a patient and a primary care provider collaborate to learn about and engage in primary care, primary care research, grant review, proposal development and advocacy. In addition, a series of educational trainings were held during the study in conjunction with national primary care conferences, international webinars and local symposia. Smith et al49 found that developing patient education material using a participatory design methodology, together with patients, clinicians, researchers and designers working as co-designers following a structured process map, is most productive and in line with a person-centred care philosophy, with a strong focus on partnership and equality.

Van der Weijden et al50 conducted a 12-month development and consensus study to develop patient-directed knowledge tools related to clinical practice guidelines. They showed that an 8-step guidance was needed to reach consensus. The authors describe minimal criteria for (1) the team composition, (2) setting the scope, (3) identifying needs, (4) the content and format, (5) testing the draft, (6) finalizing and approval, (7) dissemination and application, and lastly, (8) ownership and revision. Archambault et al51 recommends that to increase patient engagement in patient-oriented emergency medicine research, they need to have an overarching positive recommendation to support the patient engagement. The authors propose seven policy-level recommendations for the association of emergency physicians to support the creation of a national patient council with the aim to develop, adopt and adapt training material, guidelines, and tools for patient engagement, and to support increased patient engagement in emergency medicine research. Lastly, they provide nine pragmatic recommendations about engaging patients in the preparatory, execution, and translational phases of emergency medicine research.

As for translational research in biomedical research, Boenink et al52 suggest that patients should be enabled to (1) put forward their experiential knowledge, (2) develop a rich view of what an envisioned innovation might look like and to (3) connect their experiential knowledge with the envisioned innovation. The authors have therefore developed a method called “Voice of patients”, which is successful in mobilizing patients’ experiential knowledge, stimulating their imaginaries of the innovation under the discussion and to some extent, also in connecting these two. It is argued, however, that because scientists and patients frequently presuppose that patients first need is to be educated before any meaningful communication about research is possible, patients become “pseudo-professionals”, which goes against the major reason to involve patients in research: to harvest and use their experiential knowledge. Meaningful patient involvement therefore requires that the difference between scientific and clinical expertise, and patients’ experiential knowledge is acknowledged and made productive, instead of erased.53

Burke et al54 found that the presence of active medical conditions in the hospital that made decision-making difficult, prior experiences with hospital readmission, relative level of caregiver support, and pressure to make a decision quickly were important contextual themes in a qualitative study evaluating patient decision-making regarding post-acute care.

Health participants valued the perspective that patient and public involvement could bring in mental health and learning disabilities research, as shown in the study by Paul and Holt,55 but indications of frustration with tokenistic approaches to the additional involvement work was also found. In addition, the authors identified cultural and attitudinal barriers to integrate patient and public involvement across the full research process.

Graffigna et al6 show that the therapy for promoting effective patient engagement is first to fertilize a patient engagement ecosystem. It reveals that a holistic and complex approach is needed to solve underlying causes to engage patients and the public in research in healthcare. In addition, patient engagement measurement should be a routine.

Validated measures of patient engagement can fulfil several purposes, as they may constitute a powerful communication and advocacy tool to give a voice to patients and their families. Eventually, it is the only way to ensure personalization of intervention and the incorporation of patients and family caregivers’ perspectives in the design of research and innovations. Also clinicians and researchers must be engaged, actively share and discuss scientific literature, seminars, workshops, conferences. In that regard, continuing and distance education are fundamental tools with which to make patient engagement a shared goal of clinicians and researchers, rather than being a prescription to comply with. Lastly, it benefits from the family caregiver boost. Partnering with the caregivers and the family will be an important step towards increasing and ensuring the most effective patient engagement in research.

Last but not least, the European Patients Academy on Therapeutic Innovation (EUPATI)56 has developed guidelines on patient involvement in research and development.57 EUPATI has set up structures to develop and disseminate accessible, well-structured, comprehensive, scientifically reliable, and user-friendly educational material for patients on the process of medicines research and development. They argue that once armed with a deeper and better understanding of patients, patient experts, and patient advocates, it will be easier to work more effectively with the relevant authorities, healthcare professionals and industry. This in turn will positively affect medicine development processes, benefitting patients and society. The qualitative secondary analysis conducted by Hamilton et al58 using in-depth interviews with patient research partners revealed that patients experience the collaboration and work with researchers generally as positive. Eight themes emerged to be important for patients: (1) procedural requirements, (2) convenience, (3) contribution, (4) support, (5) team interaction, (6) research environment, (7) feel valued, and (8) benefits. Linking these themes together formed a conceptual framework, called PEIR, that can help explain the phenomenon of meaningful patient engagement research.

Specific Age Groups

The literature reveals some important insights as concerns specific age groups. For example, Clorafi24 showed that older adults need to have a positive relationship with the provider meeting the patient’s needs, and the distribution of a meaningful summary at the end of the provider visit in order to have clear take-away messages. Menichetti et al59 examined the design, development and optimization of a theoretically driven intervention program (PHEinAction) to increase patient engagement in older chronic populations, and showed that it is important to consider emotional, psychological, and behavioral processes to support patient engagement among older patients. In addition, Persson et al60 showed that involving children and adolescents in mental health treatment in outpatient and community mental health clinics in Sweden is evaluated as positive and negative. Young people’s recommendations for improving practice in mental healthcare was categorized as improving the (1) accessibility, (2) being heard and seen, and (3) the usefulness of sessions.

Akubuiro37 advocates that increasing the awareness associated with benefits of usage of new technologies and policy directives that establishes technological requirements are recommended initiatives to motivate millennials to participate. Another example is the study by Heffernan et al61 who show that a youth-adult partnership model in youth mental health systems research, the McCain Youth-Adult Partnerships (Y-APs) initiative, can be used to engage youth in decisions that affect them in a way that draws on their unique skills and expertise. Flexible engagement, multifaceted mentorship, reciprocal learning and authentic decision-making have led to successful partnerships providing multiple opportunities for growth for all those involved.

Systematic Reviews on Impact and Return on Investment of Patient Engagement in R&I

Limited studies have reported on the impact or return on investment of patient engagement in research and innovation. As put forward by Jennings et al,35 future research should develop a standardized measure of collaborative research impact and (cost-)effectiveness, and conduct patient and public involvement in research to evaluate the impact. For patient with rheumatology, Hamilton et al7 have developed a framework for advancing the reporting of patient engagement, based on 30 publications related to patient engagement in this line of research. Three main categories that were developed include: (1) Who (Who engages in the research, who are the patient research partners?), (2) How (How do these patients/caregivers engage in research?), and (3) When (When during a research project do these patients/caregivers engage?). These categories should be considered to be reported in order to have a better idea about the effectiveness and impact of patient engagement in research.

In a systematic review conducted by Evans et al62 on the extent, quality and impact of patient and public involvement in antimicrobial drug development research, only one relevant protocol paper published between 1996 and 2016 was identified. Despite strong policy guidance encouraging patient and public involvement at international and national levels, and anecdotal accounts of patient engagement taking place, evidence for the extent, quality and impact of patient and public involvement continues to be very scarce.

Most studies have described the importance of patient engagement and how this can be achieved, and what could be the outcomes, but the impact or return on investment has only seldomly been studied.35 For example, meaningful patient engagement in the development of medicines during the life cycle of a product requires active participation of all stakeholders and a clear understanding of respective expectations.63 Despite its importance, Boudes et al63 show that no stakeholder has a clear view on how to engage with patients in a meaningful way. The authors raise attention to the fact that there are educational gaps, and consider structure and guidance for patient engagement as highly needed. Effective collaboration requires consensus on roles, responsibilities and expectations to synergize efforts to deliver meaningful patient engagement in research and innovation and medicines life cycle.

As Pushparajah64 explains, while all stakeholders agree on the fact that patient perspectives should be taken into account in the research and innovation of therapies, interventions and medicines, the lack of standardized best practices and metrics has made it challenging to achieve consistency and measure success in patient engagement. The Union Chimique Belge (UCB) has therefore developed an internal model for patient group engagement, incorporating four key principles that are essential elements for effective collaborations based on shared ambition, transparency, accountability, and respect.

Metrics to Evaluate the Impact/ Return on Investment of Patient Engagement in R&I

In the reviewed studies, the most mentioned metric to evaluate the impact and return on investment of patient engagement in research and innovation is recognizing patients as experts with important contributions, thereby empowering them in well-informed healthcare decisions and respecting the rights of citizens in healthcare policy. Furthermore, the goal that most of the guidelines, recommendations and discussions include is the development of more patient-centred, trustworthy and effective guidelines that lead to improved implementation and quality of care.

For example, Kristensen et al65 showed that patient-reported outcome measures were included in the treatment of patients diagnosed with depression and schizophrenia using an iterative co-creation process between patients and healthcare professionals. Zhang et al66 followed the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to integrate patient values and preferences in healthcare decision-making of practice guideline development. The GRADE “Evidence-to-Decision framework” that was developed provides an empirical strategy to find and incorporate values and preferences in guidelines by performing systematic reviews and eliciting information from guidelines panel members and patient representatives. However, there is a need for more well-conducted research in order to be able to use evidence-based advice.

Adams et al36 have shown in their Steps-methodology that there are other metrics to assess impact of patient engagement in every step. In the first step (No engagement; 0–1 year), the metrics are initial contact/meetings, core partners identified, and community needs/goals identified. In the second step (Preliminary Engagement; 1–2 years), the metrics are partnership planned, and initial pilot work under way. In the third step (Initial Partnership; 2–4 years), the metrics are tangible products of collaboration (funding, data, training) realized, and community advisory board facilitating work. In the fourth step (Extended Partnership; >4 years), the metrics are extended academic services to students, increased capacity of community/employment in research, and increased connections to other investigators and potential projects. Finally, in step five (Full Partnership > 5 years), the metrics are broad recognition of academic-community partnership, broader impact on community health, publications, tenure, multiple shared grants, student training, and increased community funding.

Kreindler and Struthers67 developed a score-sheet for tangible effects of patient participation (STEPP) to assess the organizational impact of patient involvement. The items assess the magnitude of each recommendation or issue brought forward by patients, the extent of the organization’s response, and the apparent degree of patient influence on this response. The composite scores appeared to credibly reflect the degree of organizational impact and were associated with salient features of the involvement initiatives. Furthermore, participants described the STEPP as easy to use and useful for monitoring and accountability purposes. Ree et al68 conducted a systematic review to assess to what extend patient involvement is measured in patient centeredness scales for health professionals and found that there is a lack of patient centeredness scales focusing on direct and proactive involvement of patients in quality improvements. They argue that it would be useful to develop such instruments to further study the role of patient involvement in quality improvements in healthcare. They could also be used as important tools in quality improvement interventions.

Devonport et al69 showed that it is very important to assess the way you involve patients and the public in the development of health research. According to their study results, practitioners and researchers should first of all ensure clarity as concerns the patients’ and public involvement and resolve differences in aims and priorities through clear communication. Secondly, they should support relevant disclosure whilst managing risk and safety. At the same time, they should balance alongside the ethical principles of respecting patients’ autonomy and confidentiality. Thirdly, from the earliest planning phase onwards they should pay attention to relational dynamics, particularly perceived power and to the methods used to communicate as a means to minimize tacit mixed messages. Fourthly, they should create opportunities to share and establish difference as a valued component of the research process. Finally, practitioners and researchers should acknowledge constraints and limitations so these can be addressed in due time.

Patients can also be involved to develop recruitment strategies. In that regard, patients provide valuable insights when involved to optimize and target recruitment, for example to gain better insights into patient opinions for clinical trial information materials, to develop more user-friendly clinical trial websites, to define best time to recruit patients for patient-reported outcome measures, to develop clinical trial decision aids, and to develop the study brand to increase recruitment and retention (see also Jennings et al35 for an overview). Subsequently, patient and public engagement in developing methods or entire study designs can increase the probability for relevant research, population-specific sensitivity, validity and ethics of the method and the research designs.

Additionally, it can also serve to ensure relevant education and information that can help reduce health disparities, to develop guidelines on how to conduct research and to identify patient-important outcomes.32 None of the studies actually assessed the impact patient engagement had, conducted (cost-) effectiveness analyses, or tested if this was a valuable return on investment.


With the increasing call for a patient engagement revolution, further tools need to be added to the mix to include patient engagement more actively in all different processes of research and innovation on brain-related diseases. The current study showed results of a systematic review that aimed to examine patient engagement in research and innovation, predominantly investigating the literature on brain research, in order to make evidence-based recommendations about the use of patient engagement in future research in brain diseases. The healthcare industry is undergoing a paradigm shift from a traditional medical model with a fee-for-service payment system to an approach where patients are centric and dimensions such as quality of life are becoming more important.29 Patients are participating more actively in healthcare decisions-making that are related to their own health, concerning services that are based on patient-specific needs and preferences, but also on quality of healthcare provision, which is defined by patient outcome beyond pathophysiological measures to inform value-based reimbursement. In parallel, an increased emphasis on evaluating treatment benefit using patient-focused outcomes has arrived, but has not yet been investigated. Patient engagement can improve research and innovation by helping researchers select meaningful outcomes, increase social acceptability of studies, and design knowledge translation strategies that target patients’ needs.51 More specific, the Global Patient Reported Outcomes for Multiple Sclerosis (PROMS) Initiative20,70 is an international initiative led by the European Charcot Foundation and the Italian Multiple Sclerosis Society as leading agency on behalf of the MS Movement, that aims to meet the above gaps, not only in the Medicine Lifecycle area.20

As Domecq et al71 have clearly shown in their systematic review assessing patient engagement in research, there is a lack of research dedicated to identifying the best methods to achieve patient engagement although it is strictly needed. Patient engagement in healthcare research is likely to be feasible in many settings in brain-related research, but this engagement comes with a cost and can become tokenistic. Most included studies in the systematic review by Domecq et al71 have been conducted in the beginning of research (agenda setting and protocol development) and less commonly during the execution and translation of research.

In addition, Wilson et al29 identify threefold needs: Firstly, there is a need for clearly described, and evidence based, methods for guidance on how to engage patient and public in research and innovation initiatives in all the different steps of the full process. Secondly, there is a need for the development of minimum quality criteria for the development, content, and governance of patient engagement. Thirdly, clear methodologies to assess the impact and (cost-) effectiveness of patient engagement in research and innovation is needed (Wilson et al, 2018).29 The different reviewed studies clearly demonstrate that all levels in the process are important to involve patients, but also in the development of research focus, the development of research design, recruitment, data generation, data processing, and research dissemination.43

Next, clear and constructive communication, and professionally managing all stakeholders is very important. This includes communicating principles of engagement, definitions of stakeholder types, key considerations for planning, conducting and disseminating engaged research, potential engagement activities, and examples of promising practices. Adams et al36 argue that is very important to build trust over time, develop and communicate about mutually beneficial outcomes, and construct clear metrics for assessing impact. As the Step guidelines have shown, patient engagement is a long process, and is influenced by institutional ideologies, professional attitudes and patient readiness to accept new and engaging roles.8 Eventually, most studies experienced the involvement as resource and time intensive, but found it rewarding because it was the key for the formulation of the final guidelines.32

It’s important to note that most studies used the term “involvement” and only in the latest years the term patient “engagement” is receiving increasing attention within the health research field, as it reflects the collective aspiration to build a health research and care able to make patients and families active participants and co-producers of their health.

Menichetti et al59 mapped the use of different terms related to the process of giving patients a protagonist role in their own care and clarify the possible boundaries between terms that are often mixed (ie, patient engagement, patient activation, patient involvement, patient participation, patient adherence, patient empowerment and patient compliance). The definitions and the historical trend of all these terms showed the presence of specific characteristics and differences between apparently similar concepts. The indiscriminate use of all these terms reflects a lack of clarity of what researchers and healthcare systems need to do to achieve the important goal of making patients protagonists of their care.

The term patient engagement appears promising because it offers a broader systemic conceptualization of the patients’ role in research and healthcare.

In line with the MULTI-ACT model,72 patient engagement in research and care is intended as the implementation of a science developed with patients’ input to make patient engagement effective and meaningful for all the relevant stakeholders.

In sum, evidence shows that involving and engaging patients in all different stages of the process of brain research and innovation is important to improve research and healthcare provision. In addition, professionally and constructively managing and communicating with patients about the expectancies of the involvement and engagement is necessary. Limited evidence is found for the effect and impact of patient involvement and engagement in research, specifically with view to brain-related research domains. Due to the important limitations of the included studies, firm conclusions based on this systematic review cannot be drawn. Future research should aim to conduct multi-center, large sample-sized, double-blinded and detailed randomization procedures, comparable control groups, and clearly stated impact measurements, to assess (cost-) effectiveness of patient involvement and engagement in brain research and innovation.


The literature review found that the majority of studies focus on discussing guidelines and recommendations for engaging patients in healthcare and/or clinical research process rather than engaging patients in the governance of wider research and innovation programs. However, much of the data on the actual involvement/engagement of patients in healthcare research analysed by this review has already been used and will be further used as a basis for developing innovative solutions to engage patients also in the governance of research and innovation. Building on the above gaps and challenges, the MULTI-ACT project73 offers a model to capture the experiential knowledge of patients and make it scientifically relevant for all the stakeholders,72,74 moving from a “patient centric” approach toward a “patient at the table and in a team with all the other stakeholders” one. A key MULTI-ACT driven innovation has been the adoption of an Engagement Coordination Team76 in charge of managing the engagement of the Multiple Sclerosis community and enabling the integration of the great diversity of the patient experiential knowledge in the activities of the initiative, ensuring an impact on what really matter to the people affected by the disease (the final beneficiary of research), and thus a return on engagement for all the Health stakeholders involved.


EUPATI, European Patients Academy on Therapeutic Innovation; GRADE, Grading of Recommendations Assessment, Development and Evaluation; ICCPE, Italian Consensus Conference on Patient Engagement; MS, multiple sclerosis; PCORI, Patient-centred Outcomes Research Institute; PPI, patient and public involvement; PROMS, Patient Reported Outcomes for Multiple Sclerosis Initiative; PRO, Patient Reported Outcome; R&I, Research and Innovation; RCTs, randomized controlled trials; RRI, Responsible Research and Innovation; Rubric, Patient-Centred Outcomes Research Institute Engagement Rubric; STEPP, score-sheet for tangible effects of patient participation; UCB, Union Chimique Belge, Y-APs, Youth-Adult Partnerships.


This research has received funding from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the specific grant agreement no. 787570 (MULTI-ACT). Special thanks to the all the consortium and Work Package n.1 members, and to Roberta Guglielmino for useful discussion and for the help in the manuscript revision.


The authors report no conflicts of interest in this work.


1. Holmes Rovner M, French M, Sofaer S, Shaller D, Prager D, Kanouse D. A new definition of patient engagement: what is engagement and why is it important? Washington, DC: Center for Advancing Health; 2010. Available from: Accessed May 2, 2022.

2. Fisher ES, Shortell SM, Savitz LA. Implementation science: a potential catalyst for delivery system reform. JAMA. 2016;315(4):339–340. doi:10.1001/jama.2015.17949

3. O’hara JK, Lawton RJ. At a crossroads? Key challenges and future opportunities for patient involvement in patient safety. BMJ Qual Saf. 2016;25(8):565–568. doi:10.1136/bmjqs-2016-005476

4. Ghisoni M, Wilson CA, Morgan K, Edwards B, Simon N, Celia J. Priority setting in research: user led mental health research. Res Involv Engagem. 2017;3:4. doi:10.1186/s40900-016-0054-7

5. Hazo JB, Brunn M, Wykes T, McDaid D, Dorsey M, Demotes-Mainard J, Obradors-Tarragó C. European mental health research resources: Picture and recommendations of the ROAMER project. Eur Neuropsychopharmacol. 2019;29(2), 179–194. doi:10.1016/j.euroneuro.2018.11.1111

6. Graffigna G, Barello S, Riva G, Savarese M, Menichetti J, Castelnuovo G, Bertoni A. Fertilizing a patient engagement ecosystem to innovate healthcare: Toward the first Italian consensus conference on patient engagement. Front Psychol. 2017;8:812. doi:10.3389/fpsyg.2017.00812

7. Hamilton CB, Leese JC, Hoens AM, Li LC. Framework for advancing the reporting of patient engagement in rheumatology research projects. Curr Rheumatol Rep. 2017;19(7):38. doi:10.1007/s11926-017-0666-4

8. Marlett N, Shklarov S, Marshall D, Santana MJ, Wasylak T. Building new roles and relationships in research: a model of patient engagement research. Qual Life Res. 2015;24(5):1057–1067. doi:10.1007/s11136-014-0845-y

9. Graffigna G, Barello S, Bonanomi A, Lozza E. Measuring patient engagement: development and psychometric properties of the Patient Health Engagement (PHE) scale. Front Psychol. 2015;6:274. doi:10.3389/fpsyg.2015.00274

10. Abma TA, Broerse JE. Patient participation as dialogue: setting research agendas. Health Expect. 2010;13(2):160–173.

11. Espay AJ, Bonato P, Nahab FB, Maetzler W, Dean JM, Klucken J, Reilmann R. Technology in Parkinson’s disease: challenges and opportunities. Mov Disord. 2016;31(9):1272–1282. doi:10.1002/mds.26642

12. Hollis C, Morriss R, Martin J, Amani S, Cotton R, Denis M, Lewis S. Technological innovations in mental healthcare: harnessing the digital revolution. Br J Psychiatry. 2015; 206(4):263–265. doi:10.1192/bjp.bp.113.142612

13. Pii KH, Schou LH, Piil K, Jarden M. Current trends in patient and public involvement in cancer research: a systematic review. Health Expect. 2019;22(1):3–20. doi:10.1111/hex.12841

14. Silva BM, Rodrigues JJ, de la Torre Díez I, López-Coronado M, Saleem K. Mobile-health: a review of current state in 2015. J Biomed Inform. 2015;56:265–272. doi:10.1016/j.jbi.2015.06.003

15. Anastasiadou D, Folkvord F, Lupiañez‐Villanueva F. A systematic review of mHealth interventions for the support of eating disorders. Eur Eat Disord Rev. 2018;26(5):394–416. doi:10.1002/eat.23286

16. Anastasiadou D, Folkvord F, Brugnera A, Cañas Vinader L, SerranoTroncoso E, Carretero Jardí C, Lupiañez‐Villanueva F. An mHealth intervention for the treatment of patients with an eating disorder: A multicenter randomized controlled trial. Int J Eat Disord. 2020;53(7):1120–1131. doi:10.1002/erv.2609

17. Andreu-Perez J, Poon CC, Merrifield RD, Wong ST, & Yang GZ. Big data for health. IEEE J Biomed Health Inform. 2015;19(4):1193–1208. doi:10.1109/JBHI.2015.2450362

18. Zafra-Tanaka JH, Goicochea-Lugo S, Villarreal-Zegarra D, Taype-Rondan A. Characteristics and quality of clinical practice guidelines for depression in adults: a scoping review. BMC Psychiatry. 2019;19(1):76. doi:10.1186/s12888-019-2057-z

19. D’Amico E, Haase R, Ziemssen T. Review: patient-reported outcomes in multiple sclerosis care. Mult Scler Relat Disord. 2019; 33:61–66. doi:10.1016/j.msard.2019.05.019

20. Zaratin P, Vermersch P, Amato MP, Brichetto G, Coetzee T, Comi G; PROMS Initiative Working Groups. The agenda of the global patient reported outcomes for multiple sclerosis (PROMS) initiative: Progresses and open questions. Mult Scler Relat Disord. 2022;61:103757. doi:10.1016/j.msard.2022.103757

21. Armstrong MJ, Rueda JD, Gronseth GS, Mullins CD. Framework for enhancing clinical practice guidelines through continuous patient engagement. Health Expect. 2017;20(1):3–10. doi:10.1111/hex.12467

22. Ruco A, Nichol K. Patient engagement in research and innovation: a new framework. J Med Imaging Radiat Sci. 2016;47(4):290–293. doi:10.1016/j.jmir.2016.10.008

23. Rieckmann P, Boyko A, Centonze D, Elovaara I, Giovannoni G, Havrdova E, LeLorier J. Achieving patient engagement in multiple sclerosis: A perspective from the multiplesclerosis in the 21st Century Steering Group. Mult Scler Relat Disord. 2015;4(3):202–218. doi:10.1016/j.msard.2015.02.005

24. Colorafi KJ. Patient-Centered Health Information Technology: Engagement With the Plan of Care Among Older Adults With Multi-Morbidities [dissertation]. Arizona State University; 2015.

25. Hawthorne KH, & Richards L. Personal health records: a new type of electronic medical record. Records Manag J. 2017;27(3):286–301. doi:10.1108/RMJ-08-2016-0020

26. GIN Public Working Group. GIN Public toolkit: patient and public involvement in guidelines. Guideline International Network; 2015. Available from: Accessed May 2, 2022.

27. Liberati A, Altman DG and Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339(jul21 1):b2700–b2700. doi:10.1136/bmj.b2700

28. Sheridan S, Schrandt S, Forsythe L, Hilliard TS, Paez KA. Advisory panel on patient engagement (2013 inaugural panel). The PCORI engagement rubric: promising practices for partnering in research. Ann Fam Med. 2017;15(2):165–170. doi:10.1370/afm.2042

29. Wilson H, Dashiell-Aje E, Anatchkova M, Coyne K, Hareendran A, Wyrwich K. Beyond study participants: a framework for engaging patients in the selection or development of clinical outcome assessments for evaluating the benefits of treatment in medical product development. Qual Life Res. 2018;27(1):5–16. doi:10.1007/s11136-017-1577-6

30. Rae-Grant A, Day GS, Marrie RA, Rabinstein A, Cree BAC, Pringsheim T. Practice guideline recommendations summary: disease-modifying therapies for adults with multiple sclerosis: report of the guideline development, dissemination, and implementation subcommittee of the American Academy of neurology. Neurology. 2018;90(17):777–788. doi:10.1212/WNL.0000000000005347

31. Rieckmann P, Centonze D, Elovaara I, Giovannoni G, Havrdová E, Ben-Amor AF; Members of the MS in the 21st Century Steering Group. Unmet needs, burden of treatment, and patient engagement in multiple sclerosis: a combined perspective from the MS in the 21st century steering group. Mult Scler Relat Disord. 2018;19:153–160. doi:10.1016/j.msard.2017.11.013

32. Köpke S, Giordano A, Veronese S, Christin Rahn A, Kleiter I, Solari A. Patient and caregiver involvement in the formulation of guideline questions: findings from the European Academy of Neurology guideline on palliative care of people with severe multiple sclerosis. Eur J Neurol. 2019;26(1):41–50. doi:10.1111/ene.13760

33. Morbey H, Harding AJE, Swarbrick C, et al. Involving people living with dementia in research: an accessible modified Delphi survey for core outcome set development. Trials. 2019;20(1):12. doi:10.1186/s13063-018-3069-6

34. Murtagh MJ, Minion JT, Turner A, Wilson RC, Blell M, Burton PR. The ECOUTER methodology for stakeholder engagement in translational research. BMC Med Ethics. 2017;18(1):24. doi:10.1186/s12910-017-0167-z

35. Jennings H, Slade M, Bates P, Munday E, Toney R. Best practice framework for Patient and Public Involvement (PPI) in collaborative data analysis of qualitative mental health research: methodology development and refinement. BMC Psychiatry. 2018;18(1):213. doi:10.1186/s12888-018-1794-8

36. Adams A, Williamson A, Sorkness C, Hatfield P, Eggen A, Esmond S. The steps model: a practical tool for engaging communities to improve health outcomes. Acad Med. 2017;92(6):890. doi:10.1097/ACM.0000000000001677

37. Akubuiro JO A Quantitative Study of the Factors Affecting Health Care Technology Use in the Millennial Generation [Doctoral dissertation]. Northcentral University; 2018.

38. Arauwou JA Older Adults’ Perceptions of the UTAUT2 Factors Related to Intention to use a Patient Portal for Engagement in their Healthcare [Doctoral dissertation]. Northcentral University; 2017.

39. Gabel N, Altshuler DB, Brezzell A, Briceño EM, Boileau NR, Hervey-Jumper SL. Health related quality of life in adult low and high-grade glioma patients using the national institutes of health Patient Reported Outcomes Measurement Information System (PROMIS) and neuro-QOL assessments. Front Neurol. 2019;10:212. doi:10.3389/fneur.2019.00212

40. Robillard JM, Feng TL, Rosen A. When patient engagement and research ethics collide: lessons from a dementia forum. J Alzheimers Dis. 2017;59(1):1–10. doi:10.3233/JAD-161285

41. Blackwell RW, Lowton K, Robert G, Grudzen C, Grocott P. Using experience-based co-design with older patients, their families and staff to improve palliative care experiences in the emergency department: a reflective critique on the process and outcomes. Int J Nurs Stud. 2017;68:83–94. doi:10.1016/j.ijnurstu.2017.01.002

42. Grant S, Hazlewood GS, Peay HL, et al. Practical considerations for using online methods to engage patients in guideline development. Patient. 2018;11(2):155–166. doi:10.1007/s40271-017-0280-6

43. Nguyen T, Palisano RJ, Graham I. Perspectives and experiences with engaging youth and families in research. Phys Occup Ther Pediatr. 2019;39(3):310–323. doi:10.1080/01942638.2018.1496966

44. Baines RL, Regan de Bere S. Optimizing patient and public involvement (PPI): identifying its “essential” and “desirable” principles using a systematic review and modified Delphi methodology. Health Expect. 2018;21(1):327–335. doi:10.1111/hex.12618

45. Simblett S, Greer B, Matcham F, et al. Barriers to and facilitators of engagement with remote measurement technology for managing health: systematic review and content analysis of findings. J Med Internet Res. 2018;20(7):e10480. doi:10.2196/10480

46. Rashid A, Thomas V, Shaw T, Leng G. Patient and public involvement in the development of healthcare guidance: an overview of current methods and future challenges. Patient. 2017;10(3):277–282. doi:10.1007/s40271-016-0206-8

47. Samalin L, Honciuc M, Boyer L, et al. Efficacy of shared decision-making on treatment adherence of patients with bipolar disorder: a cluster randomized trial (ShareD-BD). BMC Psychiatry. 2018;18(1):103. doi:10.1186/s12888-018-1686-y

48. Sand J, Felzien M, Haeme R, Tapp H, Derkowski D, Westfall JM. The North American Primary Care Research Group’s Patient and Clinician Engagement Program (PaCE): demystifying patient engagement through a dyad model. Fam Pract. 2017;34(3):285–289. doi:10.1093/fampra/cmx027

49. Smith F, Wallengren C, Öhlén J. Participatory design in education materials in a health care context. Action Res. 2017;15(3):310–336. doi:10.1177/1476750316646832

50. van der Weijden T, Dreesens D, Faber MJ, et al. Developing quality criteria for patient-directed knowledge tools related to clinical practice guidelines. A development and consensus study. Health Expect. 2019;22(2):201–208. doi:10.1111/hex.12843

51. Archambault PM, McGavin C, Dainty KN, et al. Recommendations for patient engagement in patient-oriented emergency medicine research. CJEM. 2018;20(3):435–442. doi:10.1017/cem.2018.370

52. Boenink M, van der Scheer L, Garcia E, van der Burg S. Giving voice to patients: developing a discussion method to involve patients in translational research. Nanoethics. 2018;12(3):181–197. doi:10.1007/s11569-018-0319-8

53. van de Bovenkamp HM, Trappenburg MJ. Reconsidering patient participation in guideline development. Health Care Anal. 2009;17(3):198–216. doi:10.1007/s10728-008-0099-3

54. Burke RE, Jones J, Lawrence E, et al. Evaluating the quality of patient decision-making regarding post-acute care. J Gen Intern Med. 2018;33(5):678–684. doi:10.1007/s11606-017-4298-1

55. Paul C, Holt J. Involving the public in mental health and learning disability research: can we, should we, do we? J Psychiatr Ment Health Nurs. 2017;24(8):570–579. doi:10.1111/jpm.12404

56. European Patients’ Academy on Therapeutic Innovation (EUPATI). Available from: Accessed May 2, 2022.

57. Spindler P, Lima BS. Editorial: the European Patients Academy on Therapeutic Innovation (EUPATI) guidelines on patient involvement in research and development. Front Med. 2018;8(5):310. doi:10.3389/fmed.2018.00310

58. Hamilton CB, Hoens AM, Backman CL, et al. An empirically based conceptual framework for fostering meaningful patient engagement in research. Health Expect. 2018;21(1):396–406. doi:10.1111/hex.12635

59. Menichetti J, Libreri C, Lozza E, Graffigna G. Giving patients a starring role in their own care: a bibliometric analysis of the on-going literature debate. Health Expect. 2016;19(3):516–526. doi:10.1111/hex.12299

60. Persson S, Hagquist C, Michelson D. Young voices in mental health care: exploring children’s and adolescents’ service experiences and preferences. Clin Child Psychol Psychiatry. 2017;22(1):140–151. doi:10.1177/1359104516656722

61. Heffernan OS, Herzog TM, Schiralli JE, Hawke LD, Chaim G, Henderson JL. Implementation of a youth-adult partnership model in youth mental health systems research: challenges and successes. Health Expect. 2017;20(6):1183–1188. doi:10.1111/hex.12554

62. Evans D, Bird E, Gibson A, et al. North Bristol microbiology patient panel. Extent, quality and impact of patient and public involvement in antimicrobial drug development research: a systematic review. Health Expect. 2018;21(1):75–81. doi:10.1111/hex.12587

63. Boudes M, Robinson P, Bertelsen N, et al. What do stakeholders expect from patient engagement: are these expectations being met? Health Expect. 2018;21(6):1035–1045. doi:10.1111/hex.12797

64. Pushparajah DS. Making patient engagement a reality. Patient. 2018;11(1):1–8. doi:10.1007/s40271-017-0264-6

65. Kristensen S, Mainz J, Baandrup L, et al. Conceptualizing patient-reported outcome measures for use within two Danish psychiatric clinical registries: description of an iterative co-creation process between patients and healthcare professionals. Nord J Psychiatry. 2018;72(6):409–419. doi:10.1080/08039488.2018.1492017

66. Zhang Y, Coello PA, Brożek J, Wiercioch W, Etxeandia-Ikobaltzeta I, Schünemann HJ. Using patient values and preferences to inform the importance of health outcomes in practice guideline development following the GRADE approach. Health Qual Life Outcomes. 2017;15(1):52. doi:10.1186/s12955-017-0621-0

67. Kreindler SA, Struthers A. Assessing the organizational impact of patient involvement: a first STEPP. Int J Health Care Qual Assur. 2016;29(4):441–453. doi:10.1108/IJHCQA-01-2015-0013

68. Ree E, Wiig S, Manser T, Storm M. How is patient involvement measured in patient centeredness scales for health professionals? A systematic review of their measurement properties and content. BMC Health Serv Res. 2019;19(1):12. doi:10.1186/s12913-018-3798-y

69. Devonport TJ, Nicholls W, Johnston LH, Gutteridge R, Watt A. It’s not just ‘what’ you do, it’s also the ‘way’ that you do it: patient and public involvement in the development of health research. Int J Qual Health Care. 2018;30(2):152–156. doi:10.1093/intqhc/mzx177

70. The global patient reported outcomes for MS (PROMS) Initiative. Available from: Accessed May 2, 2022.

71. Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89. doi:10.1186/1472-6963-14-89

72. Zaratin P, Bertorello D, Guglielmino R, et al. The MULTI-ACT model: the path forward for participatory and anticipatory governance in health research and care. Health Res Policy Syst. 2022;20(1):22. doi:10.1186/s12961-022-00825-2

73. MULTI-ACT Project. A collective research impact framework and multi-variate models to foster the true engagement of actors and stakeholders in health research and Innovation. Available from: Accessed May 2, 2022.

74. Kork AA, Antonini C, García-Torea N, et al. Collective health research assessment: developing a tool to measure the impact of multistakeholder research initiatives. Health Res Policy Syst. 2022;20(1):49. doi:10.1186/s12961-022-00856-9

75. MULTI-ACT project - partnerships. Available from: Accessed May 2, 2022.

76. MULTI-ACT deliverable D1.6. Final version of the MULTI‐ACT patient engagement in health R&I guidelines. Available from: Accessed May 2, 2022.

Creative Commons License © 2022 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.