Importance of hemodialysis-related outcomes: comparison of ratings by a self-help group, clinicians, and health technology assessment authors with those by a large reference group of patients

Background

For some years now, more and more emphasis has been placed on patient-centered health care as well as on active patient involvement in clinical research and health technology assessment (HTA).^1–4

The Institute of Medicine (IOM) defined patient-centered care as “Providing care that is respectful of and responsive to individual patient preferences, needs, and values, and ensuring that patient values guide all clinical decisions.”⁵

The need for patient engagement in health care research is widely recognized, and measures are being undertaken in many areas to promote further engagement.^3,6 One area of interest is the definition and rating of patient-relevant outcomes. However, the decision as to which outcomes are important enough to be evaluated in clinical research or HTA is still largely made by clinicians or HTA authors; patients are only occasionally involved in this process.

It is unclear to what extent the importance that clinicians or HTA authors assign to certain outcomes reflects the importance assigned by patients. Several studies have found discrepancies between preferences of patients and those of other stakeholders.^7,8

Moreover, if patients are involved, this normally happens in an unstructured manner by consultation with a small group of patient representatives (eg, members of a self-help group). It is unclear to what extent they represent the perspective of the total population of patients with a specific condition.

Hemodialysis is a time-consuming and long-term intervention for the treatment of chronic kidney disease (CKD) and affects many different outcomes. While there are obvious ones, such as mortality, morbidity and health-related quality of life (HRQoL), others related to the impact on daily living or the provision of health care (eg, journey time to dialysis or its duration) may also be important.⁹

We had previously conducted a survey of 4518 CKD patients treated by hemodialysis to investigate the relative importance that patients assign to various outcomes and found that the 3 most important ones were the safety of treatment, HRQoL and satisfaction with care.¹⁰ However, the involvement of such a large patient sample in outcome prioritization in clinical research or HTA would not normally be feasible in practice.

We therefore aimed to investigate to what extent the preferences of 3 stakeholder groups normally or occasionally involved in outcome prioritization in clinical research or HTA deviate from the preferences of a large reference group of patients. For this purpose, we compared the preferences of members of a self-help group for CKD, nephrologists and HTA authors with those of a reference group with CKD in respect of the importance of hemodialysis-related outcomes.

Participants and methods

Stakeholder groups

Reference group: CKD patients from a quality management program in nephrology

The reference group consisted of CKD patients treated by hemodialysis in the centers of the Board of Trustees for Dialysis and Renal Transplant (Kuratorium fuer Dialyse und Nierentransplantation [KfH]), the largest nonprofit provider of hemodialysis in Germany. It comprises >200 units treating ~18,000 patients per year and sends an annual survey to all of its patients as a part of its quality management program “Quality in Nephrology” (QiN).¹¹ Our questionnaire was included in the survey of 2008. A detailed description has been published elsewhere.¹⁰

CKD patients from a self-help group

As patient representatives, we included patients recruited from a regional self-help group for CKD, the Lower-Rhine Section of the Kidney Interest Group (Sektion Niederrhein, Interessengemeinschaft Niere e.V.). A paper-and-pencil version of the questionnaire was distributed during a regional meeting of this group; in addition, the group’s office was provided with paper-and-pencil versions to distribute in further meetings. The office also sent the link of the online version of the questionnaire to all members of the self-help group with known email addresses.

Nephrologists

We identified nephrologists by selecting the largest city of each of the 16 German federal states and identified outpatient dialysis centers in these cities via a Google search. We contacted a total of 80 centers that had a website with an email address. We sent them a link of the online questionnaire and asked them to participate and also to forward the link to further nephrologists. In addition, a nephrologist collaborating with our project disseminated the link in national nephrology meetings.

HTA authors

A total of 25 HTA authors included in the Institute for Quality and Efficiency in Healthcare database of external experts or personally known to the authors of this paper were contacted via email and provided with a link of the online questionnaire. They were asked to participate and also to forward the link to further HTA authors.

Questionnaire

Outcomes for inclusion in the questionnaire were identified using patient input; details are described elsewhere.¹⁰ In short, a group discussion was held with patients from a self-help group to identify the important outcomes. A pretest was conducted with 5 patients, not otherwise involved in the project, to assess the comprehensibility of the questions and the completeness of the outcomes used.

Subsequently, 23 hemodialysis-related outcomes were identified and added to a routinely conducted annual survey (“Reference group: CKD patients from a quality management program in nephrology” section). We only considered patient-relevant outcomes, defined as how a patient feels, functions or survives.¹² Surrogate outcomes, that is, outcomes not directly perceptible by patients (eg, laboratory values), were not considered.

On the basis of a systematic literature search for instruments for assessing preferences, we chose a rating scale in the form of a discrete visual analog scale (VAS) to rate the preferences for the 23 outcomes.⁷ This scale consisted of 9 categories, ranging from “not important” (1) to “very important” (9). A complete list of the 23 outcomes and their definition in the questionnaire is listed in Table 1. Patients from the reference group and the self-help group were asked to indicate how important the outcomes were to them by assigning a score from 1 to 9; similarly, nephrologists and HTA authors were asked to indicate how important they thought that these outcomes were to patients.

Table 1 Outcomes included in the questionnaire Note: aTranslated from German; patient version (nephrologists and HTA authors were asked how important the goals and requirements were for patients). Abbreviations: GI, gastrointestinal; HRQoL, health-related quality of life; HTA, health technology assessment.

Sociodemographic data were collected from all 4 groups. In addition, we collected information on diabetes diagnosis, time on dialysis in years, and education, and occupation, from the patients. From nephrologists, we collected information on work experience, the average number of patients treated and the work setting (hospital or practice).

Patients, nephrologists and HTA authors were informed about the purpose of the questionnaire and were asked to give informed consent. The paper-and-pencil versions were handed out together with a consent form and a return envelope. In the online version, the participants were requested to check a box to confirm their informed consent before proceeding with the questionnaire. The data of the entire questionnaire were collected within a quality assurance framework, and a specially assigned data protection commissioner ensured that patient data were dealt with correctly according to German data protection laws. Only data of patients who gave written informed consent were considered.

Data analysis

Descriptive statistics (mean scores and standard deviation) were used to calculate the ratings of the importance of the 23 outcomes. The results from the QiN survey¹⁰ were used as a reference to which the other groups were compared (reference group). Comparisons with this group were performed in the following 2 different ways:

(1)

In the first analysis, we compared the 7 top-ranked outcomes. The number of outcomes evaluated and reported is usually restricted both in clinical research and in evidence synthesis. While there is no general international consensus on the exact number of outcomes to be reported, the Cochrane Collaboration and the Grading of Recommendations Assessment, Development and Evaluation working group report up to 7 outcomes.13,14 Outcomes not included in this list are not usually considered in the weighing of benefits and harms of an intervention.

(2)

In the second analysis, we compared the extent of deviations in the rankings when all 23 outcomes were considered. We compared the rankings of the 3 stakeholder groups for each outcome with those of the reference group, counting how often the outcomes differed by 0–2, 3–5, 6–8 or >8 ranks. In addition, we compared the 7 top-ranked outcomes of the 3 stakeholder groups with each other, without considering the reference group.

Results

Participants

In the QiN study, we had previously analyzed 4518 questionnaires from hemodialysis patients. In this study, we received 49 completed questionnaires from the self-help group (Table 2). While most characteristics were similar in the 2 patient groups, patients from the self-help group were on average younger, had diabetes less often and had spent more time on dialysis. They were also more often employed and were more likely to have higher education. We received 18 completed questionnaires from HTA authors and 19 from nephrologists (Table 3). On average, HTA authors had 6 years of HTA experience and nephrologists had 20 years of experience in treating patients with CKD and, at the time of the survey, were each responsible for 40–150 patients.

Table 2 Characteristics of patient groups Note: *Independent t-test.

Table 3 Characteristics of HTA authors and nephrologists Notes: aOne works both as a clinician and as a research associate. bOther degrees include: economics, geography, dietetics and psychology. Abbreviations: CKD, chronic kidney disease; HTA, health technology assessment.

Ranking of outcomes

Comparison of the 3 stakeholder groups with the reference group

Variations in the 7 top-ranked outcomes

Detailed results of the importance ratings (mean and standard deviation) are listed in Table S1. Figure 1 shows the 7 top-ranked outcomes of all 4 groups. The following 3 outcomes were ranked within the top 7 by all groups: safety, HRQoL and emotional state.

Figure 1 Comparison of outcomes. Notes: Lines link outcomes that are within the top 7 outcomes in all 4 groups. The top 7 outcomes of the reference group are marked in gray. Abbreviations: GI, gastrointestinal; HRQoL, health-related quality of life; HTA, health technology assessment.

Self-help group versus reference group: 5 outcomes were ranked within the top 7 outcomes by both groups. Both groups ranked safety in the first place. However, the self-help group rated physical functioning (rank 2 versus rank 11) and traumatic punctures (rank 5 versus rank 9) as being more important than did the reference group. In contrast, hospital stays and accompanying symptoms were rated by the reference group as being more important.

Nephrologists versus reference group: 4 outcomes were ranked within the top 7 outcomes by both groups. Both groups had a similarly high ranking for satisfaction with care (rank 2 versus rank 3). However, nephrologists rated access to nursing staff (rank 1 versus rank 8), physical functioning (rank 5 versus rank 11) and nausea/drop in blood pressure (rank 6 versus rank 10) as being more important than did the reference group. In contrast, hospital stays, accompanying symptoms and hemodialysis duration were rated by the reference group as being more important.

HTA authors versus reference group: 4 outcomes were ranked within the top 7 outcomes by both groups. Both groups had a similarly high ranking for HRQoL (rank 1 versus rank 2). However, HTA authors rated work (rank 2 versus rank 23), life expectancy (rank 3 versus rank 15) and physical functioning (rank 5 versus rank 11) as being more important than did the reference group. In contrast, satisfaction with care, accompanying symptoms and hemodialysis duration were rated by the reference group as being more important.

Deviations when all 23 outcomes are considered

Figure 2 shows the extent of deviations in the rankings of all outcomes for each of the 3 stakeholder groups compared to the ranking by the reference group. In the comparison of the rankings by the self-help group with those by the reference group, 18 outcomes were ranked with very small or small differences and 5 outcomes were ranked with large or very large differences.

Figure 2 Extent of differences in outcome rankings between the self-help group, nephrologists and HTA authors versus the reference group. Notes: Each bar represents 1 of the 3 groups versus the reference group. Colored fields indicate how many outcomes were ranked differently to the rankings of the reference group, while numbers in the fields indicate the quantity of outcomes this applies to. The first bar starts with 10 outcomes ranked with very small differences (0–2 ranks) by the self-help group versus the reference group. The next field shows that 8 outcomes were ranked with small differences (3–5 ranks) versus the reference group. Abbreviation: HTA, health technology assessment.

In the comparison of the rankings by the nephrologists with those by the reference group, 17 outcomes were ranked with small or very small differences and 6 outcomes were ranked with large or very large differences.

In the comparison of the rankings by the HTA authors with those by the reference group, 13 outcomes were ranked with small or very small differences and 10 outcomes were ranked with large or very large differences (even a difference of 21 ranks for 1 outcome, work).

Comparison of the self-help group, nephrologists and HTA authors

Figure 3 shows the 7 top-ranked outcomes of the 3 stakeholder groups. As stated, they agreed on 4 outcomes (safety, physical functioning, HRQoL and emotional state). Outcomes that were ranked within the top 7 solely by one group were traumatic punctures and hemodialysis duration (self-help group), access to nursing staff and nausea/drop in blood pressure (nephrologists), as well as work, life expectancy and hospital stays (HTA authors).

Figure 3 The 7 top-ranked outcomes of stakeholder groups. Note: Shaded outcomes are also among the 7 top-ranked outcomes of the reference patient group. Abbreviations: HRQoL, health-related quality of life; HTA, health technology assessment.

Discussion

The aim of this study was to investigate to what extent the preferences of 3 stakeholder groups (CKD patients from a self-help group, nephrologists and HTA authors) normally or occasionally involved in outcome prioritization in clinical research and HTA deviate from the preferences of a large reference group of patients with CKD. Only 3 outcomes, which were safety, HRQoL and emotional state, were consistently ranked within the top 7 outcomes by all 4 groups. Depending on the group asked, the outcomes included in the top 7 varied considerably. This is problematic as often only a single stakeholder group is involved in outcome prioritization in clinical research and HTA.^15,16

Although the rankings of the self-help group were more similar to those of the reference group than the rankings of clinicians and HTA authors, there were notable deviations, which may be explained by the fact that survey results are affected by sampling bias if participants are selected nonrandomly,^17–19 as was the case with the self-help group in our study, who, for instance, were younger than patients in the reference group. However, an exploratory analysis comparing patients of both groups within similar age strata did not yield results that were more similar to each other; factors other than age, therefore, seem to have led to different results. Specific factors that differ from those in the general patient population might be difficult to identify and even more difficult to control for. This has to be taken into account when considering preferences elicited only from patients in a specific subgroup.

The 2 outcomes ranked highest by the nephrologists were access to nursing staff and satisfaction with care at the dialysis clinic, indicating that clinicians place more emphasis on outcomes related to clinic organization than do patients and HTA authors.

Outcomes ranked quite differently by HTA authors than by the reference patients include life expectancy and the possibility to work despite dialysis; these were rated to be far more important by HTA authors. The fact that life expectancy (rank 15) was not considered by the reference patients to be as important as other outcomes might be explained by the fact that CKD is no longer an acute life-threatening condition. The possibility to work (ranked last) is understandably not so important for a patient group that is largely retired. When evaluating the importance of outcomes, the HTA authors were probably influenced by their past experiences concerning which outcomes can be reliably measured in clinical studies.

Strengths and limitations

The strength of our study is the possibility to compare the importance of outcomes for 3 stakeholder groups normally or occasionally involved in the process of outcome prioritization with the results of a large reference group of patients whose preferences are not usually available.

We chose a discrete VAS to enable the elicitation of preferences for a large number of outcomes. Other methods, such as conjoint analysis, analytical hierarchy process and utility assessment methods, are also used for this purpose, but as they are not suitable for dealing with the large number of outcomes analyzed in our study, they were not feasible alternatives.^7,15 Although a ceiling effect was observed, we were able to establish a ranking of outcomes and to compare the ranks between the different samples.

Nonrepresentative samples (such as patients from a self-help group) do not necessarily provide exact estimates of the preferences of a large patient population. However, in decision-making in clinical research and HTA, the input of health care professionals is not based on a representative sample either.

Comparison with previous research

Studies using preference elicitation methods have been published increasingly over the last decade.^7,16,20 In our previous descriptive review on methods to prioritize outcomes, we identified several studies comparing preferences for the outcomes of different stakeholders; most of them compared preferences of patients and clinicians.⁷ The studies identified were conducted in different health care settings and used different methods to evaluate the importance of outcomes. Most studies showed notable differences between preferences of patients and other stakeholders, that is, either different hierarchies or at least different strengths of preferences for outcomes.

We did not identify a single study comparing the preferences of any of the 3 stakeholder groups, and investigated with such a large reference group as the one included in our analysis; furthermore, we are not aware of studies comparing the preferences of HTA authors or dialysis patients with those of other groups.

Preference elicitation methods, especially multicriteria decision analysis (MCDA), are being increasingly used for reimbursement decisions in HTA.^20,21 A systematic review of studies applying MCDA methods in decisions addressing the trade-off between costs and benefits found that the majority of studies used health outcomes as a criterion to elicit preferences from different stakeholder groups. Interestingly, health policy decision makers were more intensely involved, while patient involvement was limited.¹⁵

Implications of research findings

The choice of important outcomes has implications for the evaluation of efficacy in clinical research and consequently of comparative effectiveness in HTAs. The explicit elicitation of preferences from different stakeholder groups can make the process of clinical research and HTA more transparent and add important dimensions.^22,23

The question arises as to how outcomes reflecting patient preferences can be identified for decision-making in clinical research and HTA in a valid and feasible manner. Our study indicates that even though some deviations exist, the preferences of a self-help group might represent an adequate proxy for a large patient population with the same condition.

The preferences of clinicians and HTA authors deviated to a greater extent from those of the reference group but not so much that they should be discarded in cases where direct access to patient groups is not feasible. More importantly, their insights are relevant for outcomes not perceptible but still potentially important to patients (eg, clinical outcomes). Furthermore, they can advise on outcomes that might not be of major importance to the individual patient but to the health care system or society in general.

The Standardized Outcomes in Nephrology-Hemodialysis (SONG-HD) initiative is working on a core outcome set for all trials in hemodialysis involving patients and health professionals.²⁴ However, further research is needed to analyze how preferences of different groups can be implemented in research decisions.²⁵

Conclusion

Our study indicates that preferences of different stakeholders (CKD patients from a self-help group, nephrologists and HTA authors) differ to a varying extent from those of a large reference group of patients with CKD. The involvement of self-help groups might be a feasible means of including patient preferences in outcome prioritization, but the effects of the mode of sampling have to be considered in the interpretation of results. In addition, the preferences of health care professionals such as clinicians and HTA authors should be considered, as only the combination of all preferences forms the basis of a transparent discussion on important outcomes in clinical research and HTA.

Acknowledgments

The authors thank Natalie McGauran for medical writing support. We acknowledge support for the Article Processing Charge by the Deutsche Forschungsgemeinschaft and the Open Access Publication Fund of Bielefeld University.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.		Facey K, Boivin A, Gracia J, et al. Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26(3):334–340.
2.		Facey KM, Hansen HP. Patient-focused HTAs. Int J Technol Assess Health Care. 2011;27(4):273–274.
3.		Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89.
4.		Muhlbacher AC, Juhnke C. Patienten- und Bürgerpartizipation in der Entscheidungsfindung im Gesundheitswesen insbesondere bei der Bewertung von Arzneimitteln [Involving patients, the insured and the general public in healthcare decision making]. Z Evid Fortbild Qual Gesundhwes. 2016;11(0–111):36–44. German.
5.		Institute of Medicine (U.S.); Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
6.		Ryan M, Kinghorn P, Entwistle VA, Francis JJ. Valuing patients’ experiences of healthcare processes: towards broader applications of existing methods. Soc Sci Med. 2014;106:194–203.
7.		Janssen IM, Gerhardus A, Schroer-Gunther MA, Scheibler F. A descriptive review on methods to prioritize outcomes in a health care context. Health Expect. 2015;18(6):1873–1893.
8.		Muhlbacher AC, Juhnke C. Patient preferences versus physicians’ judgement: does it make a difference in healthcare decision making? Appl Health Econ Health Policy. 2013;11(3):163–180.
9.		Urquhart-Secord R, Craig JC, Hemmelgarn B, et al. Patient and caregiver priorities for outcomes in hemodialysis: an international nominal group technique study. Am J Kidney Dis. 2016;68(3):444–454.
10.		Janssen IM, Gerhardus A, von Gersdorff GD, et al. Preferences of patients undergoing hemodialysis – results from a questionnaire-based study with 4,518 patients. Patient Prefer Adherence. 2015;9:847–855.
11.		Stoffel MP, Barth C, Lauterbach KW, Baldamus CA. Evidence-based medical quality management in dialysis – part I: routine implementation of QiN, a German quality management system. Clin Nephrol. 2004;62(3):208–218.
12.		Biomarkers Definitions Working G. Biomarkers and surrogate endpoints: preferred definitions and conceptual framework. Clin Pharmacol Ther. 2001;69(3):89–95.
13.		Higgins J, Green S [homepage on the Internet], editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. 2011. Available from: www.cochrane-handbook.org. Accessed November 9, 2016.
14.		Schünemann H, Brożek J, Oxman A, editors. GRADE Handbook for Grading Quality of Evidence and Strength of Recommendation. Version 3.2 [updated March 2009]. The GRADE Working Group; 2009. Available from: http://www.cc-ims.net/gradepro.
15.		Wahlster P, Goetghebeur M, Kriza C, Niederlander C, Kolominsky-Rabas P, National Leading-Edge Cluster Medical Technologies ‘Medical Valley EMN. Balancing costs and benefits at different stages of medical innovation: a systematic review of Multi-criteria decision analysis (MCDA). BMC Health Serv Res. 2015;15:262.
16.		Gargon E, Gurung B, Medley N, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One. 2014;9(6):e99111.
17.		Kalsbeek W, Heiss G. Building bridges between populations and samples in epidemiological studies. Annu Rev Public Health. 2000;21:147–169.
18.		Tyrer S, Heyman B. Sampling in epidemiological research: issues, hazards and pitfalls. BJPsych Bull. 2016;40(2):57–60.
19.		Neumark DE, Stommel M, Given CW, Given BA. Research design and subject characteristics predicting nonparticipation in a panel survey of older families with cancer. Nurs Res. 2001;50(6):363–368.
20.		Marsh K, IJ M, Thokala P, et al. Multiple criteria decision analysis for health care decision making-emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19(2):125–137.
21.		Angelis A, Kanavos P. Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics. 2016;34(5):435–446.
22.		Gagnon MP, Desmartis M, Lepage-Savary D, et al. Introducing patients’ and the public’s perspectives to health technology assessment: a systematic review of international experiences. Int J Technol Assess Health Care. 2011;27(1):31–42.
23.		Hailey D, Werko S, Bakri R, et al. Involvement of consumers in health technology assessment activities by Inahta agencies. Int J Technol Assess Health Care. 2013;29(1):79–83.
24.		Tong A, Manns B, Hemmelgarn B, et al. Establishing core outcome domains in hemodialysis: report of the Standardized Outcomes in Nephrology-Hemodialysis (SONG-HD) Consensus Workshop. Am J Kidney Dis. In press 2016.
25.		Gagnon MP, Desmartis M, Gagnon J, et al. Framework for user involvement in health technology assessment at the local level: views of health managers, user representatives, and clinicians. Int J Technol Assess Health Care. 2015;31(1–2):68–77.

Supplementary material

Table S1 Ratings of importance of outcomes by the stakeholder groups Abbreviations: GI, gastrointestinal; HRQoL, health-related quality of life; HTA, health technology assessment.