Selecting Performance Indicators and Targets in Health Care: An International Scoping Review and Standardized Process Framework

Introduction

Over the past 20 years, governments and health care agencies have mandated the collection and monitoring of hundreds of indicators by health service providers, such as hospitals.^1,2 Indicators are defined as “measurable elements of practice performance” that relate to clinical, population health, financial, or organizational performance.³ In the USA, the National Quality Forum (NQF) approved indicator list has grown from 200 in 2005 to over 700 in 2011.⁴ In Canada, over 300 quality indicators have been reported by Ontario hospitals.⁶ Health system managers in the USA and Canada, as well as the UK and Australia, submit that the emergence of over-measurement has negative consequences.^4–6 Arbitrary, top-down approaches to mandating the collecting and monitoring of indicators continue to contribute to over-measurement and data that do not necessarily reflect local context and stakeholder needs.^7–10 Large volume of measures can paralyze decision-making.^1,11 The development of indicators without local input creates a lack of trust between providers and political bodies, and invites the gaming of metrics given organizations may economically benefit from higher comparative rankings.^6,9 The building of the information technology and data infrastructure required to support measurement has amplified the amount of data available, complicated decision-making and increased the financial cost of data collection to health care organizations.⁴

These findings have led to calls for a more balanced approach to measurement, focusing on how indicators advance strategic goals and user-value.^4,11 The World Health Organization urged providers to prioritize measures that align with the specific information needs of those who use indicators for improvement.⁷ The Institute of Medicine, National Quality Forum, Canadian Institute for Health Information, and Statistics Canada completed indicator review exercises and recommended reducing the number of indicators monitored by health system providers.^12–14 Research papers also share indicator selection processes in areas like emergency medicine and primary care.^9,15,16 These reports describe different methods used to select indicators at the system or clinical service level. Despite these calls, inconsistent, arbitrary approaches to selecting indicators and targets may lead to variable quality and a lack of engagement that could prohibit those responsible for improving performance from taking action.^1,5,7,9,11

Study Purpose

The following paper describes a scoping review to answer the question, “How and by whom are health care performance indicators and targets selected in Commonwealth Fund countries?” The review synthesizes different approaches used to select health care indicators and targets and proposes a standardized indicator selection process framework.

Methodology

A scoping review was completed following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guideline.^17,61 PubMed and Web of Science search engines were utilized given their focus on biomedicine and health care, and coverage of multiple databases. Inclusion criteria consisted of articles published from 2010 to 2020, written in English, with a focus on acute care hospital services. Articles from the 11 countries in the Commonwealth Fund’s annual comparison of health system outcomes (www.commonweathfund.org) were included. These countries, comprised of Australia, Canada, France, Germany, The Netherlands, New Zealand, Norway, Sweden, Switzerland, the United Kingdom, and the United States, were selected given their health systems comparability. Keywords used within the literature search are available as Supplementary Data. Exclusion criteria consisted of articles that were study protocols or systematic reviews; did not describe a selection process; involved non-hospital-based services; were not written in English; or were from non-Commonwealth Fund comparator countries.

A grey-literature search was conducted by identifying publicly available documents on government agency and health policy institutes’ websites from each of the 11 Commonwealth Fund countries. Hand searching of 24 policy health institute websites resulted in identifying 83 documents for review of which 11 were included in this review. A listing of the institutes is available as Supplementary Data.

In total, forty-four documents (thirty-three peer-reviewed and 11 grey-literature) met the criteria for final review. Figure 1 illustrates the PRISMA-SCR peer-reviewed and grey-literature search decision tree.¹⁷

Figure 1 The flow of study identification and selection according to PRISMA-ScR guidelines. Abbreviation: PRISMA-ScR, preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews. Note: Adapted from Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLOS Medicine. 2021;18(3):e1003583. Creative commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.17

Data were systematically extracted from each of the included papers and were used to inform the development of a standardized process framework. This process included identifying common themes arising from the literature and arranging them under preliminary categories.¹⁸ Initial categories included what is being selected (clinical indicators, business indicators, targets), rationale for the selection process, individuals involved in the process, steps used to prepare for the process, methods and criteria used to select indicators, and post-selection activities. The development of the framework was iterative with changes to categorization and wording as data extraction and thematic analysis progressed.¹⁸

Results

Tables 1 and 2 summarize the country of origin and field of study of included papers, respectively. Tables 3 and 4 summarize the content of the peer-reviewed and grey-literature, respectively. Five themes emerged from the analysis of peer-reviewed and grey-literature documents: aim; governance; preparation; methodologies; and validation.

Table 1 Peer Reviewed and Grey Literature by Country

Table 2 Peer Reviewed and Grey Literature by Field of Study

Table 3 Scoping Review Peer-Reviewed Literature Summary

Table 4 Scoping Review Grey-Literature Summary

Aim

The first theme addresses the rationale that an indicator and target selection process is conducted. Subthemes that arose to form this theme included describing an aim statement (100% of peer-reviewed and 100% of grey-literature documents); offering a set of principles to guide the work (30.3% of peer-reviewed and 72.7% of grey-literature documents); and identifying the system or organizational unit in which the work is based (100% of peer-reviewed and 100% of grey-literature documents).

Peer-reviewed literature focused on specific organizational units measuring discrete clinical processes or outcomes, whereas grey-literature focused on system-level indicators that address quality and patient safety. As a result, peer-reviewed papers’ aim statements are more narrowly defined than those found in the grey-literature. Values such as openness, transparency, and accountability were frequently cited as being part of a set of guiding principles.^{26,30,31,38,53,55} Papers that described selection processes within clinical areas stressed that indicators should match the care continuum, so they are representative of the patient journey and clinical practice.^23,42,46

All documents noted the system or organizational unit the process was designed to inform.^{13,14,16,19–59} Indicator selection processes must consider the intended use of the indicator given indicators can be used for a variety of reasons, including accountability, process improvement, and public reporting.^{32,36,42,47,55,56}

Governance

Governance oversight of indicator and target selection processes is the second theme. Subthemes included identifying structures that provide an oversight function (97.0% of peer-reviewed and 100% of grey-literature documents), and the identification and recruitment of process participants (93.9% of peer-reviewed and 72.7% of grey-literature documents).

Documents shared two models of governance. The first model is a single-body governance structure where the process is managed and conducted by one steering committee or expert panel.^{20–22,24,27,31–35,37,39–57} The second model is a multi-body structure that has a steering committee responsible for managing the process and offering recommendations, but also includes sub-committees or expert panels that assist with literature reviews, data collection, and assessments.^{13,14,16,19,23,25,26,29,30,38,51–53,58,59}

Most documents identified who participated in indicator and target selection processes. Several peer-reviewed papers revealed studies that involved only physicians,^{20,27,29,33,42,46,49,50} while other studies incorporated broader representation from areas, such as nursing, allied health, research, quality, and administration.^{13,14,16,19–24,26,28,31–35,37–41,43–56,58} Some indicator selection processes involved patients and family members, noting that their contribution ensured indicators connected with the consumer of services.^{14,19,21,25,38–40,44,47,51,53,55,58,59} Studies using only physicians and nurses cited their clinical backgrounds as a strength but acknowledged the need to expand participation to mitigate medical biases.^{19,29,33,46,48,50} Studies that had expert panels with broader memberships believed that broader participation enabled a more inclusive view of the care process.^{13,16,19,21,25,40,42,51,52,54,55,59} One study required panelists to have at least 2 years of clinical experience, and a balance of gender representation to ensure experience and equity perspectives are considered in the selection of indicators.⁴⁸

Preparation

Five subthemes emerged to create the third theme: preparation. These sub-themes consisted of seeking early input from end-users on their indicator needs (21.2% peer-reviewed and 36.4% grey literature documents); reviewing literature and evidence-based guidelines (87.8% peer-reviewed and 36.4% of grey-literature documents); compiling an indicator inventory and definition list (100% of peer-reviewed and 100% of grey-literature documents); placing indicators into categorical themes (84.8% of peer-reviewed and 81.8% of grey-literature documents); and, developing participant orientation and training materials (33.3% of peer-reviewed and 36.4% of grey-literature documents).

All documents described an indicator selection process that involved consulting data libraries, peer-reviewed literature, and clinical guidelines to create an inventory of potential indicators. Documents stated that a final list of indicators built from comprehensive sources improves their relevancy to end-users while enabling future comparability and benchmarking.^{13,14,16,19–59}

Documents that sought end-user input upfront on indicator knowledge and user requirements^{13,20,21,32,42,45,47,50,52,55,56} and issued orientation materials^{13,14,20,21,26,32,37,38,40,44,46,50,54,58} reported increased participant engagement and improved understanding of the process among participants.

Process Methodologies

The fourth theme speaks to the methodologies used to assess and recommend indicators and targets. This theme emerged from documents that described consensus-building methods (97.0% of peer-reviewed and 90.9% of grey-literature documents); facilitation (24.2% of peer-reviewed and 89.7% of grey-literature documents); indicator assessment criteria (100% of peer-previewed and 90.9% of grey-literature documents); and rating methods by which indicators were assessed (90.9% of peer-reviewed and 54.5% of grey-literature documents).

Studies that utilize consensus-building processes, such as a modified-Delphi approach, involved issuing surveys to seek input on the number of indicators to be considered, followed by an in-person or online web conference to finalize the selection.^{12,16,20–22,24–29,31,33–35,37,40–46,49–53,55,59} These consensus-building processes increase validity with participants^{12,16,20–22,24–29,31,33–35,37,40–46,49–53,55,59} Several papers reported that processes facilitated by a neutral expert minimized steering committee or expert panel bias.^{16,20,30,35,38,41,49} Common indicator assessment criteria include relevance, scientific soundness, feasibility, and usability, as per the Appraisal of Indicators through Research and Evaluation (AIRE) tool.^{13,14,19–60} Analytically, studies generally ranked indicators using Likert scales from 1 to 7 or 1 to 9.^{20,21,25,26,29–31,33–35,37,40,42–46,48,50} Two studies allowed participants to provide qualitative feedback on indicators between modified-Delphi rounds.^34,48

Validation

The final theme, validation, emerged in two forms: quantitatively testing for data quality (39.4% of peer-reviewed and 63.6% of grey-literature documents) and qualitative feedback from end-users on face validity (21.2% of peer-reviewed and 63.6% of grey-literature documents). Processes that statistically tested indicators for data quality emphasized the increased scientific soundness of the indicators^{14,16,19,23,25,28,30,33,36,39,41,43,47,49,54–56,58,59} and better informed the target setting.⁴³ Processes that validated a final list of indicators with end-users reported improved relevance and usability by users, especially in cases where the expert panels did not include front-line directors, managers, or patients.^{13,14,21,23,28,30,36,37,44,54–56,58,59}

Target Setting

No document summarized a process that directly addressed the setting of indicator targets or benchmarks. Literature that made suggestions in this area emphasized that targets and benchmarks need to be better defined and understood by end-users.^{23,31,36,42,43,49,50,54–57} Benchmarks have limitations as they are generally based on a subset of performance units versus an agreed upon best practice. Benchmarks are not necessarily the required target, given a unit’s indicator performance may already have exceeded the benchmark. Thus, an indicator target may be intended to simply maintain performance.^{23,31,36,42,43,49,50,54,56} Similarly, given that performance on an indicator may be behind the benchmark, incremental improvement towards the benchmark may be a more appropriate target.^54,57 Targets may also distort practice choices or not reflect the care needed at the patient level given targets generally measure macro-outcomes at the population level versus operational realities. As such, targets must be set carefully by testing for scientific soundness and relevance to end-users.^{23,31,36,42,43,49,50,54}

Discussion

This scoping review identified 44 documents that addressed the research question, “How and by whom are health care key performance indicators and targets selected in Commonwealth Fund countries?” The review demonstrates that structured indicator selection processes are generally governed by steering committees or expert panels, are guided by clear aim statements, involve literature searches on potential indicators, use consensus seeking methods, categorize indicators as process, outcome, or structure metrics, and align indicators to categories, such as strategic themes or clinical care processes. Not all documents describe preparation and validation stages. Only a few studies engaged end-users up front about how they use indicators or validated the relevance of the chosen indicators with stakeholders after indicators were selected. Similarly, only a few studies tested selected indicators for data quality. No paper directly addressed the targets, but some advocated for testing data to ensure benchmarking could occur.

Most papers focused on clinical access and quality indicators and did not address medical education, system-level, or business-related indicators in areas such as finance, human resources, and supply chain. As such, governors of indicator selection processes should be mindful that health care managers, administrative leaders and other clinical actors have many more indicators to manage than only those related to quality and patient safety.

Indicator selection processes varied in who participated, in particular, those included on expert panels. Findings seem to indicate that, given the multidisciplinary nature of health care delivery and the need to ensure indicators match the information needs of end-users, indicator selection processes should be inclusive and equitable.^7,48 No study has directly addressed how to set performance targets. Moreover, given that indicators are used as an instrument to help advance performance, findings suggest that those responsible for indicator selection and target setting should ensure end-users understand and provide input on the targets they are accountable for achieving.

While all documents described steps of an indicator selection process, no process included each component identified in the thematic analysis. Incomplete indicator selection processes risk over-measurement, the lack of prioritizing strategic and operational goals, lack of support by end-users, and paralyzed decision-making ability.^2–4,7,11 These gaps present an opportunity to build a standardized framework that can assist organizations in developing a comprehensive indicator and target selection process.

The 5-P Indicator Selection Process Framework

The themes extracted from each of the papers led to the development of a standardized process framework. The 5-P Indicator Selection Process Framework consists of five domains and 17 elements. The framework’s first domain, “Purpose”, sets out the reasons why an indicator selection and target setting process is undertaken. By stating the process aim, the principles used to guide the process, and the organization level at which the indicators will be used, organizations can facilitate a shared understanding of the rationale they are trying to achieve. The second domain, “Polity”, identifies the governance structures that manage the selection process, how the process will be resourced, and who will participate. The third domain, “Prepare”, addresses how to plan for selection. Elements include asking potential users about their experience with indicators, researching literature and best practices, developing a defined inventory of potential indicators, categorizing indicators into strategic themes, and delivering training or orientation materials and programs. The fourth domain, “Procedure”, describes the steps used to assess indicators and targets and gain consensus. Elements include consensus building methods, facilitation, assessment criteria, analytical assessment of potential indicators, and target-setting. The final domain of the framework is “Prove”. This domain describes the validation processes used to test any final set of indicators for data quality and relevance with end-users. Table 5 summarizes each domain and element.

Table 5 The 5-P Indicator Selection Process Framework

Whereas previously published constructs such as the Appraisal of Indicators through Research and Evaluation (AIRE) Instrument⁶⁰ and the Quality Indicator Critical Appraisal (QICA) tool⁸ suggest criteria to guide, which individual indicators should be considered, the 5-P Indicator Selection Process Framework offers a standardized process that governs and guides the overall process. Organizations that are mature in their performance measurement capabilities may use the framework to assess their current process and identify targeted opportunities for improvement. Less mature organizations and organizations undergoing transformations that may influence the number or type of indicators they measure should consider adopting the framework as a whole. By adopting the framework, organizations will have a clear purpose for selecting indicators; adopt governance models that enhance equitable participation from multiple stakeholder groups, including patients; select indicators based on evidence-based criteria; and ensure indicators match end-users needs by validating any final set of indicators.

Limitations

The scoping review focused on clinical services generally found within acute care hospital settings. Future research should include articles on primary care and post-acute care to validate or extend the proposed framework. Only one individual screened and reviewed the papers in this review. To mitigate potential biases, the reviewer regularly debriefed with other members of the research team on inclusion and exclusion decisions. The 5-P Indicator Selection Process Framework is the result of a scoping review and has not been validated in real-world settings. Future research may involve validating the framework by assessing it in practice.

Conclusion

This paper began by describing the proliferation of measurement in health care and risks associated with inconsistent indicator section processes. The overabundance of indicators has paralyzed decision-making, and eroded trust between those who ask for indicators and those who are expected to use them to make change. Many policy institutes and academics have called for a more appropriate, lower number of indicators. Indicator selection or reduction processes cannot occur by happenstance. The adoption or elimination of indicators should be guided by the 5-P Indicator Selection Process Framework to ensure a systematic, evidence-based, and inclusive approach that engages measurement experts and those who use indicators to monitor and improve performance in both selection and validation.

The 5-P Indicator Selection Process Framework provides a practical, standardized structure that health care agencies, hospitals, and clinical disciplines can use to guide the selection of performance indicators and targets. The 5-P Indicator Selection Process Framework may also act as an implementation framework by which researchers evaluate how health care agencies select indicators and targets.