Back to Journals » Patient Preference and Adherence » Volume 14

Defining Criteria for Guiding Cancer Patients to Find a Reputable Complementary Medicine Provider: Results of a Literature Review and a Consensus Procedure

Authors Rogge AA, Baur I, Blettner G, Holtkamp U, Horneber M, Jahn P, Joos S, Keberle S, Kettelgerdes A, Klemperer D, Längler A, Voiß P, Weis J, Witt CM

Received 11 September 2019

Accepted for publication 14 February 2020

Published 21 April 2020 Volume 2020:14 Pages 747—755


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Johnny Chen

Alizé A Rogge,1 Isabel Baur,2 Gabriele Blettner,3 Ulrike Holtkamp,4 Markus Horneber,5 Patrick Jahn,6 Stefanie Joos,7 Silva Keberle,8 Anita Kettelgerdes,9 David Klemperer,10 Alfred Längler,11 Petra Voiß,12,13 Joachim Weis,14 Claudia M Witt1,15,16

1Institute for Social Medicine, Epidemiology, and Health Economics, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität Zu Berlin, and Berlin Institute of Health, Berlin, Germany; 2Legal Institute, Competence Center Medicine - Ethics - Law Helvetiae, University of Zurich, Zurich, Switzerland; 3Deutsche Krebshilfe, INFONETZ KREBS, Bonn, Germany; 4German Leukemia & Lymphoma Patients‘ Association, Bonn, Germany; 5Department of Internal Medicine, Division of Oncology and Hematology, Paracelsus Medical University, Klinikum Nuremberg, Nuremberg, Germany; 6Department for Nursing Science, Institute for Health Science, Medical Faculty, Eberhard Karls University Tübingen, Tübingen, Germany; 7Institute for General Practice and Interprofessional Health Care, University Clinic Tübingen, Tübingen 72076, Germany; 8Eskamed AG, Basel, Switzerland; 9TK – Techniker Krankenkasse, Hamburg, Germany; 10Faculty of Social and Health Sciences, Ostbayerische Technische Hochschule Regensburg, Regensburg., Germany; 11Gemeinschaftskrankenhaus Herdecke, Institute of Integrative Medicine, Witten/Herdecke University, Herdecke, Germany; 12Department of Internal and Integrative Medicine, Evang. Kliniken Essen-Mitte, Faculty of Medicine, University of Duisburg-Essen, Essen, Germany; 13Gynaecological Clinic, Städtisches Klinikum Karlsruhe gGmbH, Karlsruhe, Germany; 14Department of Self-Help Research, Comprehensive Cancer Center University Clinic Freiburg, Freiburg i.Br., Germany; 15Institute for Complementary and Integrative Medicine, University of Zurich and University Hospital Zurich, Zurich, Switzerland; 16Center for Integrative Medicine, University of Maryland School of Medicine Baltimore, Baltimore, Maryland

Correspondence: Claudia M Witt
Institute for Complementary and Integrative Medicine, University Hospital Zurich and University Zurich, Sonneggstrasse 6, Zurich 8091, Switzerland

Purpose: Even in cases of positive evidence for complementary medicine (CM) therapies, it is still difficult for cancer patients to identify reputable providers. The aim of this study was to develop and evaluate a criteria list to provide guidance to cancer patients seeking a reputable CM provider.
Methods: The design combined a literature review, an expert consensus procedure (n=15) and an assessment from three stakeholder perspectives (patients (n=18), CM providers (n=26) and oncology physicians (n=20)).
Results: A total of 30 existing CM criteria were extracted from the literature, and 12 more were added by the experts. The main challenge was to define criteria that could easily be applied by the patients. A final comprehensive list of 8 criteria guiding cancer patients to find a reputable CM provider was developed.
Conclusion: Health professionals and cancer information services might find the criteria list helpful when aiming to strengthen patients’ awareness of quality-related factors associated with CM providers. The criteria developed might be helpful when standards are established for quality assurance in CM in oncology.

Keywords: neoplasms, complementary medicine, patient-centered care, standards, healthcare quality assurance


Approximately half of cancer patients use complementary medicine (CM) during their cancer treatment,1 and that percentage has appeared to increase over time.2 The term CM summarizes a broad range of heterogeneous therapies such as yoga, relaxation techniques, herbal medicine or whole traditional systems such as traditional Chinese medicine, many of which lack sufficient data or clinical evidence. A recent retrospective observational study using data from the American National Cancer Database3 suggested that CM provided by nonmedical personnel was associated with a higher risk of death due to declined cancer treatments. These findings have been vigorously challenged for the validity of its conclusion due to sampling and data issues.4 Nevertheless, the publication highlights the need to involve medical personnel to guide patients when they want to use CM. In particular, there is an increasing body of positive evidence for some therapies, such as yoga to reduce fatigue and sleep disturbances5 or mindfulness-based stress reduction (MBSR)6 to improve quality of life and mood symptoms which is reflected in existing medical guidelines.7,8

The evidence for the quality of CM offers is often difficult to judge for patients and physicians without experience in CM. The lack of standards and regulations for nonmedical CM providers (providers without a medical license) in some countries and the fact that they are usually not part of the oncology care team make this process even more difficult. Some hospitals and private practices offer in-house CM programs to ensure the quality of their interventions; however, this approach is not considered standard. Therefore, patients may sometimes feel alone when searching for reputable nonmedical providers for CM therapies. Even when patients try to follow their oncology physicians’ evidence-based CM recommendations, they are still unsure which providers offer both safety and high quality. Therefore, a need exists to discriminate between more and less “safe” nonmedical CM providers.

As part of the Competence Network Complementary Medicine in Oncology (KOKON),9 a collaborative research project in Germany funded by the German Cancer Aid (grant 70112369), this project aims to develop CM criteria as indicators of reputable CM providers in oncology.


The design (see Figure 1: Flowchart of the criteria list development process) included a systematic literature review (Phase I) to identify existing criteria for reputable CM providers. This process was followed by an international and interprofessional expert consensus procedure (Phase II) and a practice evaluation with relevant stakeholder groups (cancer patients, CM providers and oncology physicians). Experts recruited for this study were based in Germany and Switzerland, and from diverse backgrounds (medical oncology, primary care, psychology, psycho-oncology, nursing, pediatric oncology, law, statutory health insurance, patient advocacy/patient representation, research methodology, public health, and epidemiology). Oral informed consent was provided by the participating experts, patients provided written informed consent and the other survey participants were informed about the aim of the project and that if they completed the survey, their answers were anonymous.

Figure 1 Flowchart of the criteria list development process.

Phase I – Literature Review

Literature Inclusion and Exclusion Criteria

Publications were included from scientific journals, working papers, and theses. Additionally, information materials, websites and other types of publications addressing criteria and standards for reputable CM providers for cancer patients, physicians, CM providers and leaders of cancer support groups were included. Publications were excluded if the stated criteria were in the context of neither oncology nor CM, or the publication was not available in the English or German language. Publications were not restricted by year of publication.

Search Strategy

Electronic databases (Ovid MEDLINE, CENTRAL, EMBASE and Web of Science) were searched from June until August 31, 2017. We linked the following keywords and text words and, if possible, combined them with subject headings: professional field related to complementary and integrative oncology (integrative oncology, OR oncology AND complementary medicine OR integrative medicine) AND elements of quality assurance (criteria OR standard OR quality OR safety) AND/OR profession (provid* OR health professional OR physician). Our search strategy included all types of publications and research designs. In addition, members of the expert consensus procedure were asked to provide additional literature, especially gray and/or unpublished literature. Moreover, we searched the web using the same terms for websites and materials that provide information on how to choose reputable CM providers.

Selection of Studies and Data Extraction

One reviewer (AAR) searched the literature reviewing the results thoroughly by assessing titles and abstracts (if possible depending on the type of publication). Publications were excluded if they did not mention any criteria. Full text copies were generated from the resulting literature, assessed and data extracted by one reviewer (AAR). The study selection and data extraction were supervised by a methodologist (CMW). Criteria were then derived from qualitative analysis. Therefore, aqualitative content analysis according to Flick10 was performed. Two independent reviewers (AAR, CMW) extracted criteria. Content units were identified to cluster the selected criteria using inductive coding strategies. Moreover, an intersubjective validation of the coding by the two reviewers to verify the reliability of the data was conducted. Based on this, criteria themes were developed and discussed within the team (education and training, medical approach, attitude, cost and reimbursement). Criteria were categorized by both reviewers independently, and any discrepancies considering categorization were documented and resolved via discussion.

Phase II – Expert Consensus Procedure Including an Evaluation on Three Different Levels

A three-round expert consensus procedure was performed. Experts (n=15) were invited to participate in this study by email explaining the aim of this project. This group of experts discussed, developed and redefined existing criteria from the literature search in multiple rounds. In the first two rounds, online surveys were completed, followed by conference calls and written exchange to find consensus.

In a first survey, the experts indicated the importance of the criteria found in the literature on a 0–10 Numeric Rating Scale (NRS, 0 = “not important at all” to 10 = “fully important”). The summarized results were discussed in a conference call, and the criteria were revised accordingly. In the second survey, the remaining criteria were judged regarding their accessibility (open access = the information is easily accessible eg during consultation the patient can ask the provider, provider has to be asked, provider has to be observed) and relevance (NRS 0–10; 0 = “not relevant at all” to 10 = “very relevant”). Furthermore, experts indicated the necessity (mandatory, optional or unnecessary) of each criterion for patients, physicians and an optional future registry (as an obligation tool for CM providers). This registry could serve as a quality assurance system in order to identify reputable CM providers. As a predefined threshold, 50% of the experts had to find a criterion relevant to a certain group (patient, physician, registry level) to keep it on the list, and two-thirds had to indicate that the criterion was mandatory to make it necessary for the resulting criteria list. This multilayer process was followed by a second conference call finalizing the criteria for the practice evaluation.

Furthermore, an introductory text and a disclaimer were prepared to provide context to cancer patients when they read the final leaflet. This approach was approved by the expert group.

For the practice evaluation, the developed leaflet was sent to cancer patients, oncology physicians and CM providers based in Germany. Cancer patients and oncology physicians were recruited within the ongoing KOKON-KTO study (Trial registration number: DRKS00012704). CM providers were recruited by newsletters of professional associations in the field of yoga, naturopathy, acupuncture and MBSR.

Using an online survey or hard copies, 24 questions (NRS 0–10 Likert scale: 0 = “don’t agree at all” to 10 = “fully agree”) on the importance and usefulness of the criteria were asked. Open-ended fields allowed survey participants to provide additional feedback.

The evaluation results were discussed during two more conference calls with the expert group, and a final criteria list, including the introductory text and a disclaimer for cancer patients, was agreed upon.

Data from the online surveys were analyzed using descriptive statistics. The best practices in consensus methods were informed by Murphy et al.11 The study was approved by the relevant local ethics board (Ethics Committee of Charité – Universitätsmedizin Berlin (EA1/127/17)).


Literature Review

From the literature search (Phase I), 13 articles from databases and the internet describing criteria for reputable CM providers in oncology were revealed, and two more articles from the experts were included (see Figure 2: Flowchart for literature search and study selection). A total of 30 existing CM criteria were extracted. The criteria (see Table 1: criteria for reputable providers from the literature) were categorized into four dominant themes, namely, education and training,1222 medical approach,12,13,15-17,1921,2326 attitude1215,2023 and cost and reimbursement,1216,21,22 which were used as the basis for the subsequent expert consensus procedure. Most articles were developed for patient use, and only one-third of the articles addressed physicians, CM providers and others. The only detailed criteria lists addressed the training of CM providers. Criteria lists were usually developed from an insurance perspective or for memberships of professional associations. No criteria list purely supported health professionals and/or consulting services in recommending nonmedical CM providers.

Table 1 Criteria for Reputable CM Providers from the Literature

Figure 2 Flowchart for literature search and study selection.

Phase II – Expert Consensus Procedure and Evaluation

The 30 criteria resulting from the literature search were merged into 27 after the elimination of duplicate criteria. The first survey provided feedback from all experts, and 26 criteria (with the exception of one criterion, namely, membership in a professional association) were rated as highly important (median: 9–10 on the NRS; mean: 7.4–10). In the following conference call, when discussing the survey results, participants agreed upon the criteria as necessary and applicable to a German-speaking target group. Additionally, two criteria were deleted for reasons of redundancy, but 12 new criteria were added to the list.

In the second survey from the available 36 criteria, 23 were seen as mandatory for informing patients, 28 seemed relevant for physicians, and 14 were viewed as necessary for a possible registry (see Appendix 1: Results of the 2nd survey of the expert consensus procedure). Furthermore, it was decided to focus on the criteria list for guiding patients in this project. In the following conference call, the criteria were merged, rephrased and brought into an order, resulting in a list of 10 criteria. During the discussion, experts agreed that although some of the criteria are important, they are difficult to verify by patients. Furthermore, the introductory text and the disclaimer for the patient leaflet were discussed and revised, and a consented pilot version of the leaflet was developed.

In the third phase, the leaflet was evaluated by cancer patients (n = 18), oncology physicians (n = 20) and CM providers (yoga instructors (n = 10), MBSR instructors (n = 7), naturopathic physicians (n = 6), and physicians trained in acupuncture (n = 3)). Most of the CM providers (88.5%) regularly applied CM treatments, and two-thirds (65.4%) regularly treated cancer patients. In general, and independent of their background, the stakeholders agreed that the developed criteria are important. However, three aspects were mentioned: some criteria should be rephrased to be more understandable for patients, some of the criteria were seen as difficult to verify, and views differed on the number of years of experience needed for treating cancer patients. When discussing the evaluation results in the final expert conference call, experts agreed on revising the criteria for better understandability and separating the three criteria that are difficult to verify (having adequate training for the CM therapy, being experienced in treating cancer patients, and participating in continuous training) from the rest of the criteria. One criterion was split into two criteria, which resulted in a final list of 8 mandatory criteria and 3 optional criteria that are more difficult to verify. In addition, the introductory text and the disclaimer were revised (see Figure 3: Leaflet for patients without the disclaimer).

Figure 3 Leaflet for patients without the disclaimer.


A comprehensive list of 8 criteria guiding cancer patients to find a reputable CM provider was systematically developed in an international and interprofessional consensus procedure. To our knowledge, this is the first consented criteria list to provide helpful guidance to cancer patients when seeking a CM provider.

The developed CM criteria might contribute to better quality control and regulation of CM providers in oncology care, which can help to ensure safe therapies for cancer patients considering CM alongside their cancer treatment. Our approach had the advantage of combining the current information from the literature with expert knowledge to develop the resulting list of criteria. Engaging international experts from different professions with different perspectives allowed for a comprehensive approach aimed at making the criteria more applicable to all types of cancer patients with different cancer entities, therefore facilitating increased importance and generalizability. The criteria list was tested with different stakeholder groups. Two potential user populations were included (cancer patients and oncology physicians) to ensure understandability and acceptance in the main target groups. In addition, as a third stakeholder group, a range of CM providers (nonmedical CM providers and medical doctors applying CM) provided their feedback.

However, a focus on three stakeholder groups, the experts and participants involved in the evaluation might not encompass the entire spectrum of perspectives, CM professions and cancer entities and stages. Criteria were selected based on anonymous online surveys but also expert discussions, hence, a detection bias due to the characteristics of the participants or a reader bias in interpreting the results of the online surveys based on the selection of experts might be possible. Moreover, all experts and participants were based in Germany and Switzerland. Adjustments and validations might be necessary for usage in, for example, Asian or African countries. All experts were selected and invited to participate in the study by the study team. Overall they had a broad range of expertise, although, their opinions about CM varied widely in the group, all of them had addressed the topic CM before. Moreover, cancer patients included due to their participation in the KOKON-KTO study might not represent the whole scope of available patients. The literature review focused on the search terms “integrative oncology” and “complementary medicine” because they are frequently used nowadays, not searching for “alternative medicine” might be seen as a limitation.

Most criteria found in the literature are useful when discussing about the quality assurance of CM providers, but would be very difficult to verify for patients, because the information needed is not freely accessible. In the development of a list with practical implications, only criteria that could be examined by patients were included in the 8 mandatory criteria. However, as a result, all criteria addressing attitudes had to be deleted. Those criteria might be better placed in structures (eg, registries) that include a contract of obligation.15,27 Moreover, criteria such as the membership in a professional organization were seen as important. However, in many countries the providers of complementary therapies are not well regulated and because of this, it is was not seen as a broadly applicable criterion. In countries, where complementary therapies are well regulated and respective professional organizations are implemented, this can be used as additional criterion. This developed criteria list can be seen as a first step. Future developments might be criteria lists for oncology physicians or health insurance companies for use in their collaborations with CM providers. Moreover, a future registry could be established based on the CM criteria provided. Being listed in this registry could function as a quality certificate for CM providers working with cancer patients. In addition, the impact of the current list for patients on outcomes needs to be evaluated in a prospective study.

This project provides only one tool for a more informed choice of patients. To ensure overall quality of CM use in oncology, oncology physicians should be trained in giving CM advice to cancer patients and to support choosing supportive, evidence-based CM therapies.28 Moreover, other health professionals such as nurses, pharmacologists and psychooncologists should be involved when informing cancer patients on CM therapies.

Trial Registration

This project is part of the KOKON-KTO study registered as DRKS00012704 on the “German Clinical Trials Register” (date of registration: August 28, 2017).

Data Sharing Statement

The datasets analyzed during the current study are available from the corresponding author on reasonable request.


We would like to thank Brigitte Blum, Iris Bartsch, Katja Icke, all participants in the evaluation process and the scientists at the Institute for Social Medicine, Epidemiology, and Health Economics, Charité – Universitaetsmedizin Berlin, for their helpful comments on the study .

Author Contributions

AAR and CMW performed the systematic review, wrote the manuscript and contributed to the design of the study. AAR coordinated the study and analyzed the data. CMW wrote the grant application and had the overall responsibility for the study. All authors contributed to the development of the criteria list (including data interpretation), drafting and revising of the manuscript, gave final approval of the version to be published, and agreed to be accountable for all aspects of the work.


This project is part of the collaborative research project KOKON, which was supported by the German Cancer Aid [grant number 109863].


Patrick Jahn reports grants, personal fees from Chugai, personal fees from Bristol-Myers Squibb, non-financial support from Helsinn, non-financial support from Tesaro, outside the submitted work. Petra Voiß reports grants from Carstens-Stiftung, during the conduct of the study. Claudia M. Witt reports personal fees, non-financial support from Several Swiss Hospitals, outside the submitted work. The authors report no other conflicts of interest in this work.


1. Horneber M, Bueschel G, Dennert G, Less D, Ritter E, Zwahlen M. How many cancer patients use complementary and alternative medicine. Integr Cancer Ther. 2012;11(3):187–203. doi:10.1177/1534735411423920

2. Qureshi M, Zelinski E, Carlson LE. Cancer and complementary therapies: current trends in survivors’ interest and use. Integr Cancer Ther. 2018;17(3):844–853. doi:10.1177/1534735418762496

3. Johnson SB, Park HS, Gross CP, Yu JB. Complementary medicine, refusal of conventional cancer therapy, and survival among patients with curable cancers. JAMA oncol. 2018;4(10):1375. (). doi:10.1001/jamaoncol.2018.2487

4. Physicians OAoN. Response to Johnson, et al. JAMA Oncology article; 2018, Available from: Accessed January 09, 2020.

5. Cramer H, Lauche R, Klose P, Lange S, Langhorst J, Dobos GJ. Yoga for improving health-related quality of life, mental health and cancer-related symptoms in women diagnosed with breast cancer. Cochrane Database Syst Rev. 2017;1:Cd010802.

6. Haller H, Winkler MM, Klose P, Dobos G, Kummel S, Cramer H. Mindfulness-based interventions for women with breast cancer: an updated systematic review and meta-analysis. Acta Oncol. 2017;56(12):1665–1676. doi:10.1080/0284186X.2017.1342862

7. Qaseem A, Wilt TJ, McLean RM, Forciea MA. Noninvasive treatments for acute, subacute, and chronic low back pain: a clinical practice guideline from the american college of physicians. Ann Intern Med. 2017;166(7):514–530. doi:10.7326/M16-2367

8. Lyman GH, Greenlee H, Bohlke K, et al. Integrative therapies during and after breast cancer treatment: ASCO endorsement of the SIO clinical practice guideline. J Clin Oncol. 2018;36(25):2647–2655. doi:10.1200/JCO.2018.79.2721

9. KOKON. KOKON - Kompetenznetz Komplementärmedizin in der Onkologie; 2018. Available from: Accessed March 20, 2020.

10. Flick U. Doing Triangulation and Mixed Methods. Vol. 2. Los Angeles: Sage; 2018.

11. Murphy E, Black N, Lamping D, McKee C, Sanderson C. Consensus development methods and their use in clinical guideline development: a review. Health Technol Assess. 1998;2(3). doi:10.3310/hta2030

12. Kiene H, Heimpel H. Ärztliche Professionalität und Komplementärmedizin: was ist seriöses Therapieren? Deutsches Ärzteblatt. 2010;107(12):A 548–550.

13. von Bechtolsheim K Scharlatan oder seriöser Therapeut? [Web Page]; 2016. Available from: Accessed March 20, 2020.

14. Federspiel K, Herbst V. Die Andere Medizin. “Alternative” Heilmethoden für Sie bewertet. Vol. 2. Berlin: Stiftung Warentest; 2006.

15. EMR. Reglement; 2017. Available from:

16. Dachverband Komplementärmedizin. Komplementärmedizin - ein Leitfaden für Patientinnen und Patienten; 2017.

17. Dixon A. Regulating complementary medical practitioners. King’s Fund. 2008.

18. Australian Natural Therapists Association. guidance on choosing a therapist; 2017. Available from:

19. Cancer Research UK. Finding a therapist; 2017. Available from: Accessed March 20, 2020.

20. Münsteraner Kreis. Münsteraner Memorandum Heilpraktiker. Ein Statement Der Interdisziplinären Expertengruppe “Münsteraner Kreis” Zu Einer Neuregelung Des Heilpraktikerwesens; 2017.

21. Bundesärztekammer (BÄK) KB, (KBV). Selbst zahlen? Ein Ratgeber zu Individuellen Gesundheitsleistungen (IGeL) für Patientinnen und Patienten sowie Ärztinnen und Ärzte. Vol. 2. Berlin: ÄZQ; 2012.

22. Institute NC. Thinking about complementary & alternative medicine. In: Services USDoHaH, ed; 2005. Available from:

23. National Center for Complementary and Integrative Health. Credentialing, licensing, and education; 2017. Available from: Accessed March 20, 2020.

24. ahm Health Insurance. Recognition criteria for complementary & alternative therapies; 2017. Available from:

25. National Institue of Complementary Medicine. Choosing a complementary medicine practitioner; 2017. Available from: Accessed March 20, 2020.

26. Broccard N, Durrer A, Frei M. Alternativ? Komplementär? Risiken und Nutzen unbewiesener Methoden bei Krebs. Krebsliga Schweiz. 2002;27–32.

27. Nothacker MJ. Haben Naturheilkunde und Komplementärmedizin einen Platz in Leitlinien des Registers der Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften? Complementary Med Res. 2016;23(3):174–178. doi:10.1159/000446972

28. Helmer SM, Rogge AA, Fischer F, et al. Evaluation of a blended-learning training concept to train oncology physicians to advise their patients about complementary and integrative medicine (KOKON-KTO): study protocol for a prospective, multi-center, cluster-randomized trial. Trials. 2019;20(90). doi:10.1186/s13063-019-3193-y

Creative Commons License This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.

Download Article [PDF]