The professional perspective on patient involvement in the development of quality indicators: a qualitative analysis using the example of chronic heart failure in the German health care setting

Introduction

Quality indicators (QIs) have been established in many health care settings to measure, compare, and increase the quality of health care.¹ Indicators are measurable items referring to the structures, processes or outcomes of care.² The interpretation of QIs infers a judgment on the quality of care provided. QIs must comply with high quality standards and should be constructed in a careful and transparent manner by means of a systematic method.³ QIs have to be as valid and reliable as possible and they should be widely accepted for the purpose for which they are intended to be used.³ Specifically, there is a consensus that the quality criteria, which are measured by QIs, should represent a meaningful outcome for the patient and the health care system and that patients should be systematically involved in the QI development process.^4–7 Despite the broad consensus on patient-orientation of health care items that shall be measured by QIs and on patient involvement in the QI development process, there seems to be a gap between theory and practice.^8–11 Most often process indicators are used for quality measurement, which makes it difficult to locate a link to patient-relevant aspects of health care.^12,13 Given the fact that QIs may have a significant impact on the delivery of health care, eg, in the context of pay-for-performance models, it is crucial that QIs are reasonable, understandable, measurable, believable, and achievable for all persons concerned (the RUMBA criteria for quality assessment).^14,15 However, to date, there is no methodological “gold standard” of how to involve patient representatives and patients’ perspectives most efficiently in the development and application of QIs.¹³

This study aims to contribute to improving the methodology for the development and application of QIs, since the question of whether or not and, if so, to what extent QIs reflect health care issues relevant from the patients’ perspectives has not yet been examined. On the basis of the concrete example of a set of QIs included in the German National Disease Management Guideline (German NDMG) for heart failure (HF), we investigated the current state of patient involvement in the development and application of QIs from the professional perspective.¹⁶ The standardized, rigorous development procedure of NDMGs includes the participation of patient representatives to consider the patients’ perspectives.

The clinical example of HF was chosen because HF is an increasingly prevalent condition with high mortality and morbidity rates. HF symptoms have a detrimental influence on a patient’s functional status and quality of life. Medical recommendations such as weight monitoring, fluid and salt restriction, moderate physical exercise, and pharmacological therapies have a high impact on a patient’s daily living.¹⁷ In addition, the health care of patients with HF involves providers from various sectors (ambulatory care, in-patient care, palliative care, rehabilitative care, general practice care, specialized medical care, and interventional care), therefore, problems associated with cross-sectorial care may be most important to patients. The set of QIs developed for measuring the quality of care of HF patients in Germany focuses on technical parameters and on processes of care creating the most typical picture of QIs developed to date.

Specifically, the present qualitative study investigated the views of professionals, ie, researchers, methodologists, patient representatives, and health care providers involved in the development and interpretation of QIs in the German health care setting. We explored whether these experts feel that the QIs currently in use in the German health care setting represent patient-relevant aspects of quality of care or if they create their own pattern of health care issues far from patients’ needs. In addition we analyzed the potential as well as the limitations of patient involvement in the development of QIs from a professional perspective.

Material and methods

The present project is part of the QUALIPAT HEART Study (QUALity Indicators in healthcare: A qualitative analysis of PATient involvement using the example of HEART failure) funded by the German Federal Ministry of Education and Research in Germany (project number 01GY1123).¹⁸ The study includes, apart from the present manuscript, investigations of the individual, as well as collective, views of patients with HF which will be published separately. The study was reviewed and approved by the ethics committee of the Hamburg Medical Association (31 May 2012, PV4089).

Context of the study

QIs, developed in the context of the German NDMG for chronic HF, constituted a concrete example used as a basis for interviews with experts.¹⁶ The German NDMG Program is a non-profit project set up in 2002 by the German Medical Association to promote the effective delivery of health and disability services within the framework of disease management in Germany, based on best available evidence from research and practice.¹⁹

It relies on a broad-based collaborative network of experts from medical research, in- and outpatient care, opinion leaders and consumers, as well as on designing tools, to promote an evidence-based culture within the German health and disability sector. These tools include evidence-based guidelines, the circulation of the latest evidence-based news from Germany and abroad, and training. There is a formalized and transparent methodology, including the systematic inclusion of patient representatives, in place for developing guidelines and deducting QIs from these guidelines.¹⁹ Potential QIs are derived from the guidelines’ objectives and recommendations and assessed according to five criteria: their relevance to the health care system, their risk of mismanagement, the clarity of their definition, the strength of their recommendation, and their possibility to influence indicator value, risk adjustment, barriers for implementation, and disposability of data.⁵ The final set of QIs is determined using a formal consensus process (nominal group technique). This process results in guideline-specific, ratio-based, QIs (see also Table 1, left column). Fifteen experts from eleven different medical societies and one patient representative were involved in the NDMG for chronic HF resulting in 13 guideline objectives, 127 recommendations and statements, and nine QIs (see Table 1 left column).¹⁶

Table 1 Patient-relevant aspects of care, as mentioned by the interviewed experts, correlated with the final set of quality indicators for the NDMG HF Notes: aRatio-based quality indicators derived in a formal process from the NDMG HF; bas mentioned in telephone interviews. Abbreviations: NDMG, National Disease Management Guideline; HF, heart failure; ECG, electrocardiogram; NYHA, New York Heart Association; ACE, angiotensin-converting-enzyme; DMP, disease management program.

Participants

In this qualitative interview study, a total of 15 experts were interviewed. An expert is a person that possesses specialized and profound knowledge of a field of interest due to his or her profession (eg, a scientist working in the field of quality improvement or a methodologist involved in guideline development) or personal affection (eg, a patient suffering from a certain condition and representing this patient group within formalized processes).²⁰

We attempted to represent a spectrum of expertise as broad as possible, while still related to the quality of care of patients with HF. For interviewee recruitment, an a priori list of different types of required expertise was made including researchers in the field, methodologists for guideline and QI development, patient representatives and physicians from different settings, specialties and with different roles in the QI development process.

Five women and ten men were interviewed. A general practitioner, a health service researcher, a cardiologist, one patient representative who was involved in the development process of the NDMG HF, two other patient representatives involved in guideline and QI development for other clinical topics, four methodologists working for different leading German institutions for health care quality measurement, three leading personalities in the field of quality measurement in Germany, a rehabilitation scientist, and a nursing scientist agreed to participate in the study. Most of the interviewees wished to not be personally referenced regarding their statements, positions or affiliations when results of the study are published. We were interested in the experts’ personal views, which they had developed based on their previous professional experiences as opposed to the “official” and “politically correct” statements of their institutions or organizations. To assure confidentiality, we do not provide socio-demographic or professional information related to the quotes.

Data collection

Interviews were done based on a semi-structured interview guide allowing for individual inquiries and further exploration of issues introduced by the interviewee during the interview. Interview questions were pre-scripted in an open format to elicit detailed answers not anticipated by the interviewer. Further probing was done to clarify meanings and relevance to interviewees if needed.^21,22 Interview guides were developed in an interdisciplinary team, based on topics relevant to patient involvement in QI development identified in a recent systematic review.¹³

The interview guideline covered the following topics: the relevance of the set of QIs as selected in the context of the NDMG HF from experts’ and patients’ views; the appropriateness of the selection, (desirable) properties, legitimacy and self-perception of the patient representative; the quantity and quality of contributions by the patient representatives regarding the guideline and QIs; communication or language barriers between lay persons and experts. Personal beliefs and experiences of the respondents were primarily targeted. The questions varied slightly depending on whether the interviewee was a patient representative, or a clinical or methodological expert, and on whether or not the person was involved in the NDMG HF development process.

An important technical aspect spanning all interviews was the fact that QI development should be discussed in the context of guideline development. In the German QI development process, as is international practice, potential QIs were derived from the recommendations and objectives of the guideline. This is why many interviewees did not distinguish properly between guideline and QI development so that many statements applied for both guideline and QI development.

The interview length varied between 29 and 59 minutes. The interviews were conducted via telephone by NJP (psychologist, post-doctorate research fellow), digitally recorded using a Zoom H2N recorder and fully transcribed by a research assistant. The accuracy of the transcripts was checked by NJP.

Data analysis

The interviews were analyzed using qualitative content analysis.²³ This systematic procedure is used to reduce large amounts of data while preserving and extracting the main content. The analysis focused on experts’ views on patient-relevant aspects of quality of health care, as well as on the potential and limitations of patient involvement in the development of QIs. All transcripts were read several times to ensure familiarity with the material. The transcripts were broken down into fragments of analysis, which were coded using deductively and inductively formed categories. These fragments can adopt different sizes ranging from parts of a sentence to a whole paragraph. Deductive categories were derived from the study’s aims as mentioned above, previous research results and the aspects of quality of care as worded in the guideline. These deductive categories were supplemented by inductively formed categories. Inductive categories were formed during the review of the interview material. If a relevant fragment was identified, a category name was derived from this fragment and a description of the category was drafted. Due to the exploratory nature of the study, the focus was mainly on inductive category formation.

We used the same category system to code all interviews. NJP coded all the material in close consultation with EB (physician, post-doctoral research fellow). The preliminary category system was presented to the members of the QUALIPAT HEART study group and the working group “qualitative methods” of the Department of Primary Medical Care of the University Medical Center Hamburg-Eppendorf, where it was discussed and tested for its intersubjective comprehensibility. Then, the complete material was subjected to a second coding.

Results

Overall, most interviewees regarded the involvement of patients in the development of guidelines and/or QIs per se as a valuable contribution, regardless of the concrete measurable extent of their input. Experts consistently felt that patient representation had invaluable merit when it comes to assessing the relevance or the weight of QIs. Moreover, they presented continuous reminders that it is ultimately all about the patients’ health.

[…] on the other hand, including what I said earlier about patient involvement in general, it’s just important that all processes that regulate, organize or register health care do not run without patients. Because that’s whom it all comes down to. That the signal is continuously sent: we do not make decisions above your heads. I consider this politically very relevant at the moment.

Representation of patient-relevant aspects of quality of health care

Consistently, experts perceived a large discrepancy between the set of QIs currently in use and the aspects which the interviewed experts thought would really represent the quality of care for patients with chronic HF in their daily lives. The output of the current development processes are “technical”, ratio-based QIs derived from recommendations that relate to detailed management instructions addressing medical personnel. Frequently mentioned examples were performing electrocardiograms or prescribing angiotensin-converting-enzyme inhibitors. The aspects of care, that experts considered important for the patients, were categorized to be “generic” or “soft” and were perceived to oppose the guideline-derived indicators.

Table 1 depicts aspects of quality which experts regarded as relevant to patients and explicitly denominated during the interviews (right column). These are contrasted with the nine present QIs developed in the current guideline-based process (left column). The table is separated into categories of quality of care as defined within the NDMG HF (diagnostics, general treatment strategy etc). The category “other” refers to issues mentioned by the experts that were not attributable to one of the predefined categories.

Most experts acknowledged that many of these patient-relevant aspects of quality such as a satisfying doctor-patient interaction, or efficient communication lie beyond “the limits of routine documentation”, and that there is a lack of methods and instruments to properly assess most of these aspects. Many experts did not see a possibility of those aspects being measured with a reasonable amount of resources in routine care beyond research projects. They also raised the question as to how and through whom this information should be collected and processed. Overall, the majority of interviewees detected and criticized the neglect of the patient’s perspective in QI development and proposed that patient surveys, especially for measuring the quality of physician-patient communication and interaction, should be done systematically.

Challenges of patient involvement in guideline and QI development process

From the interviewees’ points of view, the selection of patient representatives for participation in a guideline or QI development process by asking patient groups to send a delegate has little to no need for optimization. Remarkably, no expert questioned the legitimacy of the patient representative. The experts acknowledged the legitimacy of a patient representative when he or she was delegated by a self-help organization that was member of a national association of patient groups (“Bundesarbeitsgemeinschaft Selbsthilfe”), which in turn has rules regarding the transparency, independency, and neutrality of its members. In turn, experts stated that the legitimacy of patient groups, delegating patient representatives, was given by the long-standing and strong organization of self-help in Germany and by their rights laid down in the German Health Insurance Act (Social Code Book V, § 140f).

The experts did not have a clear preference of whether a “real” suffering patient or a professional patient advocate is to be preferred. Experts stated that the ideal patient representative would have their own experiences with the health problem in focus, but he or she should be able to represent not only his/her own views but also the knowledge and collected experiences of his/her delegating group. At the same time, a certain amount of “professionalism”, with regard to understanding the medical interventions in discussion, their rationale and expected effects, as well as, a basic understanding of evidence-based methods, would be extremely helpful for the patient representative in order to relate to patient-relevant aspects of health care. When the experts generalized their statements to other index diseases one issue was repeatedly raised: for various diseases, especially for rare diseases, there is a lack of self-help groups with a certain level of professionalism. In some cases, groups of patients with similar experiences may substitute patient representation in the absence of formal patient groups for a certain disease. One example brought up was that patients with a gastrectomy may have several experiences overlapping with patients with gastric tumors. In addition, not surprisingly, many patients are not aware of the existence of guidelines, which some experts thought may be associated with a low level of interest in participating in guideline development. Generally, patients, who were not members of a self-help organization, are a priori excluded from the current selection process, a principal problem to which none of the experts had a solution. Moreover, experts argued that the possibility to choose among several patient representatives or to have more than one in the development group would likely increase the quality of the patient representation.

With regard to increasing the quality of the input brought into the development process by patient representatives, the interviewed experts suggested that self-help organizations should systematically collect “organizational knowledge”. This would include, for example, patients’ experiences regarding problems or requests that arise regularly, which could help inform guideline development groups about the patients’ perspectives on health care needs in a specific field. Experts emphasized that this procedure is recommended in methodological guides for patient participation such as the “patient participation” manual issued by the Agency for Quality in Medicine, which is hosting the NGDM process in Germany.¹⁹ However, several experts had the impression that the collection and dissemination of organizational knowledge was often not effective nor professional enough and should be improved. Another suggestion was that patient representatives should have the opportunity to train, eg, communication skills, and to get familiar with evidence-based medicine working principles and procedures used in a guideline/QI development team. Experts linked the lack of needed skills and training opportunities with the general problem of insufficient financing of guideline development, which complicates the implementation of more complex recruitment strategies for patient representatives and financial compensation for self-help groups or patient advocates for guideline work.

Potential and perception of patient involvement in the development of QIs

Overall, the expert assessment of patient participation in guideline/QI development, among the interviewed experts in this study, was very positive. All experts perceived it a successful process. Patient representatives were generally understood as being “the best possible approach to the collective group of patients”. A patient representative – in general – was considered to dispose of aggregated patient experiences and interests, and views the care process across different sectors and types of health care providers. Therefore, the patient representative is able to shed light on complicated problems, such as reasons for non-compliance, for which scientific evidence is not conclusive. He or she may act as a counterbalance when there are conflicting professional, financial or political interests among delegates from different professional groups within the guideline team. There was virtually no reporting of acceptance/communication problems or language barriers between patient representatives and clinical or methodological experts during the development process.

Patient representatives in the NDMG development are equitable members of the guideline group, who are also entitled to vote. This also includes the development of QIs within the guideline program. The patient representatives reported perceiving themselves as full members of the group and being treated as such by other group members. Other experts confirm this perception. The willingness to adequately discuss patient representatives’ objections or suggestions was perceived by both sides. The scope for intervention in the process of QI development, following the development of guidelines, is the same (viz rather small) for all people involved, because QIs are generally derived by means of a defined procedure from the strong recommendations and objectives of the guideline. Experts see more potential for patient representative influence in the development of the guidelines (by addition of aspects which the medical experts might not have presented), than in the deduction of QIs. In any case, important tasks for the patient representative are questioning certain politically or economically motivated proposals of medical experts (who want to make certain services of their field of expertise central aspects of the recommendations and quality control), assessing the relevance of potential QIs from the patient’s perspective and helping to correctly define the numerators and denominators of the QIs. Overall, the experts interviewed consider the current form of patient participation in the QI-development appropriate under the conditions of the present system. If no conclusive evidence exists for certain areas, the statements of the patient representative(s) (preferably several), based on the knowledge from the dispatching organization, shall be considered the “best evidence” and shall be used to formulate a “desideratum for research”. Sporadic suggestions were made for alternative ways of patient involvement. Patient involvement should be strengthened by the use of patient surveys to identify patient-relevant areas of quality (in the sense of quality deficits to work on) and as an instrument for measuring the appropriateness of QIs. This would imply a deviation from the NDMG Program (deduction of QIs from strong recommendations of the guidelines) and, therefore, would result in completely different QIs. Other suggestions related primarily to the implementation of patient surveys (in the absence of evidence in the NDMG development process and to determine the perceived objectives of the NDMG from the patient’s perspective). Workshops, advisory boards or focus groups could ensure that patient advocates’ opinions are not based solely on personal opinions, provide a forum for the opinion-forming process of patient representatives away from the “structural superiority of doctors”, and develop patient preferences more decidedly. In all of which patients, not organized in self-help groups, could also be included (eg, for minor or rare diseases) and QIs from NDMGs could be derived via different principles.

Discussion

Summary of main results

The participation of a patient representative, delegated by a corresponding patient group, as a method of patient involvement in the development of QIs, experienced widespread acceptance and was considered to be very important. Acceptance and communication problems were rarely reported. However, there seems to be a gap between what is feasible within the given processes and what would be desirable regarding patient orientation. The study shows that the QIs developed for the NDMG HF represent only a small part of expert-perceived patient relevance in terms of quality of health care. The QIs identified or desired by the experts and those contained in the NDMG HF differ significantly.

Strengths and weaknesses

To our knowledge, this is the first study investigating the experts’ views on patient involvement in QI development. It shows the professional point of view on the subject and draws an extensive, multi-perspective image of state, problems and potential of patient involvement in the QI development process. The respondents represented all roles and professions involved in the QI development process and, furthermore, also involved independent experts. Notably, the analysis included the view of three different patient representatives who had previously participated in guideline and QI development. As a basis for interviews with experts we used the concrete example of QIs developed for management of patients with HF within the German NDMG process. Thus, one could speak of respondent triangulation, ie, the collection of different data sources.²⁴ This allows a particularly nuanced view, not only on how patient involvement should be embellished from an expert’s point of view, but also on the actual balance of the process. A limitation of the study is that the experts willing to participate in the study could potentially be subject to a positive bias in respect to guideline development, QI development, and patient involvement as it is practiced to date. In addition, using a concrete example from the German NDMG may not be fully transferable to other health care settings and QI development contexts. However, German QI development processes conform to internationally applied methods and processes.¹²

Discussion of results

The study showed significant differences between the health care issues of the QIs included in the NDMG HF and those which experts considered to be important to patients. Treatment recommendations without strong evidence cannot be a basis for the development of QIs due to the formalized method of deriving ratio-based QIs as is internationally established.¹² Most of the aspects experts thought to be probably more patient-relevant than those finally selected were indeed mentioned in the guideline underlying the concrete exemplary QI set investigated. Experts’ suggestions of patient-relevant quality aspects such as the relevance of a well-functioning doctor-patient relationship, of advice and of the negative effects of waiting times (eg, for appointments with specialists) were also confirmed by previous studies.^25,26

In general, the legitimacy of the patient representation was not queried by the experts included in our study. Hardly any acceptance and communication problems between the patient representatives and other guideline group members were reported. This may relate to the overall increasing professionalization of patient representation in Germany since the health reform law of 2004 demanding systematic patient representation in several committees responsible for political decisions and expertise in the health insurance system on the federal and state level.^27,28 The main problems of patient representation perceived by the experts were a lack of: medical and/or methodological competence of the patient representatives, systematic collection of knowledge and experiences within patient groups, transfer of such knowledge to patient representatives, patient groups for rare diseases, and a low level of awareness of the existence and importance of guidelines and QIs.

These results suggest principal problems in the process of patient representation in QI development. The widely supported opinion, that clinical trials or high-quality evidence are often missing patient-relevant aspects of care, is based on the fact that these aspects of care generally do not qualify for a set of candidate QIs a priori.²⁸ This raises the question whether the current criteria for QIs should be modified or the QI sets developed, using the current method, should be supplemented with additional measures. This new, additional type of “patient-relevant QI” could be generated using data from patient surveys, focus groups, or workshops. A recent systematic review mentioned focus groups; surveys based on written questionnaires and individual interviews with patients, patient representatives or relatives of those affected, as an alternative or additive way for patient involvement besides the participation of patient representatives in expert committees.⁸ In general, there is consensus that different methodological approaches need to be applied simultaneously in order to adequately include the perspectives of different stakeholders.²⁹ With respect to the inclusion of the patients’ perspectives, resource-saving measuring instruments and survey techniques for measuring patient-relevant aspects of quality are needed. Apart from that, a stronger focus of clinical research on patient-relevant aspects would be the basic requirement for “successful” patient participation in future QI development processes. A priority pattern of patient relevant research would be helpful to target clinical research to generate scientific evidence matching the patients’ perspectives.

With regard to patient representation by recruiting an individual patient representative to participate in the QI development group, this study identified some potential for adaptation in current processes. When there are few or no patient representatives available (because the indications have no self-help representation, eg, a rapidly progressive fatal outcome or harmless, non-chronic conditions), procedures must be formalized to ensure the representation of the patients’ perspectives. For existing patient self-help organizations, recommendations for patient representatives have already been formulated.¹⁰ Furthermore, the awareness of the existence and importance of guidelines (and QIs) for exerting influence on what health care providers do when they manage a specific clinical condition, should be promoted much more in patient circles and in the community to enlarge the circle of potential contributors. Additionally, raising the financial compensation of expenses related to the participation in such development processes (eg, travel expenses) may increase the willingness of qualified and motivated patients to participate in such groups. In conclusion, this study provides starting points on how the perspective of patients may be included more effectively into the QI development process.

Acknowledgments

The authors are grateful to all interviewees for their time and cooperation. We would also like to thank Sarah Lindemann (research assistant, Department of Primary Medical Care University Medical Center Hamburg-Eppendorf) for the transcriptions. The project was funded by the German Federal Ministry of Education and Research (project number 01GY1123). Members of the QUALIPAT HEART study group in alphabetical order: Ines Baudendistel (nursing and health care researcher), Kathrin Bernateck (MSc Public Health and Administration), Eva Blozik (public health medicine/researcher), Heidrun Herzberg (social scientist/researcher), Stefanie Joos (general practitioner/researcher), Nadine Janis Pohontsch (psychologist/researcher), Martin Scherer (general practitioner/researcher), Felix Welti (lawyer/honorary judge/researcher).

Author contributions

MS, HH, SJ, FW, and EB designed the study and acquired funding for the study. MS supervised the study implementation. EB led the study implementation and contributed to the data analysis. NJP conducted and analyzed all interviews. NJP and EB drafted the first version of the manuscript together. All authors revised the manuscript critically, and read and approved the final version of the manuscript. All authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Disclosure

All authors report no conflicts of interest in this work.

---

References

1.		Campbell SM, Kontopantelis E, Hannon K, Burke M, Barber A, Lester HE. Framework and indicator testing protocol for developing and piloting quality indicators for the UK quality and outcomes framework. BMC Fam Pract. 2011;12:85.
2.		van den Heuvel H. A strategy for the implementation of a quality indicator system in German primary care. Qual Prim Care. 2011;19(3):183–191.
3.		Klazinga N, Fischer C, ten Asbroek A. Health services research related to performance indicators and benchmarking in Europe. J Health Serv Res Policy. 2011;16 Suppl 2:38–47.
4.		Joint Commission on Accreditation of Healthcare Organizations (JCAHO). Guide to quality assurance. Chicago: JCAHO; 1998.
5.		German Agency for Quality in Medicine (ÄZQ) [homepage on the Internet]. Qualitätsindikatoren – Manual für Autoren. ÄZQ Schriftenreihe Bd. 36. Available from: http://www.aezq.de/aezq/publikationen/schriftenreihe. Accessed August 26, 2014.
6.		National Institute for Quality in Healthcare (BQS) [homepage on the Internet]. QUALIFY: Instrument for the Assessment of Quality Indicators. Available from: http://www.bqs-institut.de/innovationen/qualify-instrument.html. Accessed August 26, 2014.
7.		Delnoij DM, Rademakers JJ, Groenewegen PP. The Dutch Consumer Quality Index: an example of stakeholder involvement in indicator development. BMC Health Serv Res. 2010;10:88.
8.		Campbell SM, Shield T, Roger A, Gask L. How do stakeholder groups vary in a Delphi Technique about primary mental health care and what factors influence their ratings? Qual Saf Health Care. 2004;13(6):428–434.
9.		Marshall MN, Shekelle PG, McGlann EA, Campbell S, Brooke RH, Roland MO. Can health care quality indicators be transferred between countries? Qual Saf Health Care. 2003;12(1):8–12.
10.		Reeves D, Campbell SM, Adams J, Shekelle PG, Kontopantelis E, Roland MO. Comparison of composite measures of clinical quality in primary care. Med Care. 2007;45:489–496.
11.		Hutchings A, Raine R. A systematic review of factors affecting the judgement produced by formal consensus development methods in health care. J Health Serv Res Policy. 2006;11(3):172–179.
12.		Kötter T, Blozik E, Scherer M. Methods for the guideline-based development of quality indicators – a systematic review. Implement Sci. 2012; 7:21.
13.		Kötter T, Schaefer FA, Scherer M, Blozik E. Involving patients in quality indicator development – a systematic review. Patient Prefer Adherence. 2013;7:259–368.
14.		Campbell SM, Reeves D, Kontopantelis E, Sibbald B, Roland M. Effects of pay for performance on the quality of primary care in England. N Engl J Med. 2009;361(4):368–378.
15.		Ketelaars CA. Using the RUMBA requirements in developing criteria. Nurs Qual Connect. 1994;4(2):7.
16.		Bundesärztekammer (BÄK), Kassenärztliche Bundesvereinigung (KBV), Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften (AWMF). Nationale VersorgungsLeitlinie Chronische Herzinsuffizienz – Langfassung, 1. Auflage. Version 7. 2009, zuletzt geändert: August 2013. Available from: http://www.versorgungsleitlinien.de/themen/herzinsuffizienz. Accessed August 26, 2014.
17.		Hunt SA, American College of Cardiology; American Heart Association Task Force on Practice Guidelines (Writing Committee to Update the 2001 Guidelines for the Evaluation and Management of Heart Failure). ACC/AHA 2005 guideline update for the diagnosis and management of chronic heart failure in the adult: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Update the 2001 Guidelines for the Evaluation and Management of Heart Failure). J Am Coll Cardiol. 2004;46(6):e1–e82.
18.		Datenbank Versorgungsforschung Deutschland [homepage on the Internet]. Available from: http://www.versorgungsforschung-deutschland.de/show.php?pid=786. Accessed August 26, 2014.
19.		German Medical Association (GMA), National Association of Statutory Health Insurance Physicians (NASHIP), Association of the Scientific Medical Societies (AWMF). National Program for Disease Management Guidelines Method Report. 4th edition; 2010. Available from: http://www.versorgungsleitlinien.de/methodik/reports. Accessed August 26, 2014.
20.		Gläser J, Laudel G. Experteninterviews und qualitative Inhaltsanalyse. VS-Verlag: Wiesbaden; 2010.
21.		Kvale S. Doing Interviews. Sage: London; 2007.
22.		Meuser M, Nagel U. ExpertInneninterviews – vielfach erprobt, wenig bedacht. In: Bogner A, Littig B, Menz W, editors. Das Experteninterview – Theorie, Methode, Anwendung. Wiesbaden: VS Verlag für Sozialwissenschaften; 2005:71–93.
23.		Mayring P. Qualitative Inhaltsanalyse. Grundlagen und Techniken. Beltz: Weinheim; 2010.
24.		Flick U. Triangulation. Wiesbaden: VS Verlag; 2004.
25.		AQUA. Allgemeine Methoden in der sektorübergreifenden Qualitätssicherung im Gesundheitswesen nach § 137a SGB V. Available from: https://www.sqg.de/hintergrund/methodenpapier. Accessed August 26, 2014.
26.		Jeon YH, Kraus SG, Jowsey T, Glasgow NJ. The experience of living with chronic heart failure: a narrative review of qualitative studies. BMC Health Serv Res. 2010;10:77.
27.		Meinhardt M, Plamper E, Brunner H. Beteiligung von Patientenvertretern im Gemeinsamen Bundesausschuss. Bundesgesundheitsbl – Gesundheitsforsch – Gesundheitsschutz. 2009;52(1):96–103.
28.		Matzat J. Zur Beteiligung der Selbsthilfe an Leitlinien. In: DAG SHG. selbsthilfegruppenjahrbuch 2010. Gießen; 2010.
29.		Campbell SM, Braspenning J, Hutchinson A, Marshall MN. Research methods used in developing and applying quality indicators in primary care. BMJ. 2003;326(7393):816–819.