Back to Journals » Patient Preference and Adherence » Volume 18

Stakeholders’ Involvement in Health Technology Assessment in Kazakhstan, Poland and Bulgaria

Authors Kulembekova L, Hailey D, Kulzhanov M, Tabarov A, Georgiev S, Jaworzynska M , Yanakieva AY, Kosherbayeva L 

Received 20 December 2023

Accepted for publication 8 May 2024

Published 22 May 2024 Volume 2024:18 Pages 1009—1015


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Johnny Chen

Lyazzat Kulembekova,1 David Hailey,2 Maksut Kulzhanov,1 Adlet Tabarov,3 Svetlin Georgiev,4 Magdalena Jaworzynska,5 Antoniya Yordanova Yanakieva,4 Lyazzat Kosherbayeva1,6

1Health Policy and Management Department, Asfendiyarov Kazakh National Medical University, Almaty, Kazakhstan; 2School of Information Sciences and Computer Technology, University of Wollongong, Wollongong, Australia; 3Deputy Director of the National Center for Health Development, Ministry of Health, Astana, Kazakhstan; 4Department HTA, Faculty of Public Health “prof. Tzecomir Vodenitcharov, MD, PhD, Dsc”, Medical University, Sofia, Bulgaria; 5Faculty of Economics, Maria Curie-Sklodowska University, Lublin, Poland; 6Department of Health Policy and Organization, Al-Farabi Kazakh National University, Almaty, Kazakhstan

Correspondence: Lyazzat Kosherbayeva, Asfendiyarov Kazakh National Medical University, 88 Tole Bi Street, Almaty, Kazakhstan, Email [email protected]; [email protected]

Background: In recent years, involvement of healthcare stakeholders in health technology assessment (HTA) has been discussed as helping the inclusion of social values in the decision-making process. The aim of our research was to identify and compare details from Kazakhstan, Poland and Bulgaria on their stakeholders’ involvement in the HTA process. Information was sought on their identification, responsibilities, and regulation.
Methods: We conducted a survey of seven types of stakeholders in the healthcare systems of Kazakhstan, Poland, and Bulgaria. They included patients and the public, providers, purchasers, payers, policy makers, product makers, and principal investigators. They were questioned on their involvement in the HTA process, and on the objectives of their participation.
Results: Levels of involvement of different kinds of stakeholder varied between countries, reflecting political and administrative developments. There was full or partial agreement on the objectives of stakeholder participation. All respondents agreed that representatives of the ministry of health should be involved in selection of stakeholders for HTA.
Conclusion: Progress has been made in the involvement of stakeholders, with interest in further development in all three countries.

Keywords: health priorities, health policy, stakeholder involvement, Universal health coverage, health technology assessment


In the development and reform of their health care systems, many countries have made use of health technology assessment (HTA). The purpose of HTA is to inform decision-making in order to promote an equitable, efficient, and high-quality health system.1 HTA is a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its lifecycle.1 In the definition of HTA it was noted that the overall value may vary depending on the perspective taken (reflecting implications for the patient, relatives, caregivers, and the population), the stakeholders involved, and the decision context.1 By involving stakeholders it is essential to take into account their objectives and expectations in the HTA process, informed judgments can be made about the value of specific health technologies, answering the questions of how and to what extent specific health technologies can help protect, restore, or enhance the health of specific patient groups.2 The application of HTA in many countries is still evolving and there is some uncertainty on how the principles of HTA can be adopted by stakeholders in health care services.

To ensure transparency in the HTA process, including social values and patient perspectives,3 a number of countries actively involve stakeholders (formal and informal) in HTA. There are several barriers in involving different types of stakeholders and in the HTA process it is essential to take into account their needs and expectation.4

HTA was first used in decision making in Kazakhstan in 2009,5 in Poland6 in 2005 and in Bulgaria in 2015.7 The main purpose of HTA in the three countries to indicate it is used to inform decisions regarding the health technology being assessed and its availability within the health care system. HTA processes in Kazakhstan are regulated by the National Scientific Center for Health Development (NSCHD). Moreover, HTA reports are reviewed by the joint commission on the quality of medical services (JQC) of the Ministry of health (MoH), which includes representatives of the association, the experts from MoH and the NSCHD. In 2019–2020, the Project on “The Use of Evidence-Informed Deliberative Processes for Health Benefit Package Design in Kazakhstan” was implemented, within the framework of which it was proposed to expand the composition of the JQC by involving all types of stakeholders.8,9

HTA in Poland is used in hospitals as a methodology for the evaluation of therapeutic, pharmaceutical drugs and other procedures (non-medical therapeutic interventions). Using a review of current knowledge from scientific journals and based on economic analysis, it is indicated whether a particular solution in a hospital will have a positive impact (as effectiveness of a procedure and its cost-effectiveness). Through the analysis, it is determined in a multi-year perspective how any technology will change the operation of a particular unit.

In December 2015 the National Center for Public Health and Analyses in Bulgaria established the HTA official procedures. Currently, the National Council on Prices and Reimbursement of Medical Products carries out a HTA of the medicinal products.10

Many countries engage health care stakeholders in the HTA process, complementing government processes in the health care systems.11,12 We aimed to identify the involvement of stakeholders in the HTA process and compare the experience of the three countries in application to health care


Representatives of the three countries agreed to include in the survey seven types of stakeholders of the kind identified by Concannon et al.13 These were patients and the public; providers (medical specialists or hospitals etc.); purchasers (entities responsible for the costs of health services); payers (entities responsible for reimbursement of the interventions); policy makers (representatives of the MoH or government); product makers (industry); and principal investigators (academia).

Three representatives from each of the seven stakeholder groups were selected for the survey, to provide 21 respondents for each country. In selecting these persons, preference was given to those who had been involved at least once with HTA issues.

After selection of the stakeholder respondents in the three countries, their opinion was obtained on the terminology presented by Street et al.14 This was necessary to ensure that the respondents from different health systems agreed with the definitions of stakeholders. Consensus was obtained on the following terms.

Patient « A person with a disease or disorder who is benefiting from some aspect of the healthcare system because of that condition »;

Provider (Carer/caregiver) « An individual who is a primary or secondary caregiver for a patient or usually the main health care provider»;

Stakeholder « An individual with an interest in the primary or secondary final outcome of the HTA process ».

We analyzed the work from European Patients’ Forum survey15 and chose questions that could be directed both to patients and to other stakeholders. Respondents were asked to indicate in which of the seven stages (identification, prioritisation, assessment, producing information, internal review, and diffusion/dissemination) a particular type of stakeholder might be involved.

Respondents were also asked to indicate whether the level of involvement was Full, Partial or No. In the example that follows, of responses from Kazakhstan respondents on one of the stakeholders, there was one full and four partial agreements, presented as 27.8% agreement (Table 1).

Table 1 Vision of the Respondents on Level of Stakeholder’s Involvement in the HTA Stages

Six categories of objectives of stakeholders’ participation in HTA were given to respondents to determine their vision and agreement with each of them. This will help us to understand the common objectives of stakeholder engagement in HTA from different groups of representatives and reflect them in future documents of the country.

In addition, the same respondents were asked three open questions on selection and management of stakeholders’ participation in the HTA process:

  • who should select stakeholders to participate in HTA preparation and to involve in decision-making process;
  • should regulatory (guidance) documents be prepared to attract stakeholders (for example, how to select participants, the rights of participants in the HTA process, at what stage should a certain stakeholder be involved in HTA, etc.);
  • and who should manage stakeholders’ involvement in HTA preparation and decision-making process (government or non-government).

Participants in the interviews were provided with a hard copy of the open-ended questions on which they could record their responses. The interviewer had the opportunity to ask follow-up questions for clarification, and recorded responses to these on hard copy. Before the start of the survey, the questions presented were pilot-tested among seven respondents to determine whether the question was understandable. Survey conducted on February and April 2022 in each of the three countries. The survey was conducted in Kazakhstan in Russian and Kazakh languages, depending on the preference of the respondent, in Bulgaria in Bulgarian and in Poland in Polish.

The other responses with low agreement are included in the discussion. The results were compared and discussed by the authors in online meetings.


All 21 respondents from both Poland and Bulgaria participated in the survey. There were 18 participants from Kazakhstan with two respondents for decision makers, payers and industry. Their responses on involvement in the seven HTA stages are shown in Table 1.

Identifying the Prospects for Involving Different Types of Stakeholders in HTA Process

For Providers of care, respondents from Kazakhstan gave high scores for their involvement in Diffusion and dissemination and for Identification and Internal review. Poland also indicated support for Dissemination, though with a lower score. Bulgaria had a high score for Identification.

For Payers, Kazakhstan and Poland gave high scores for Producing information and Diffusion/dissemination. Bulgaria’s highest scores were for Prioritisation and Assessment.

Kazakhstan gave high scores for Purchasers in all activities except Diffusion and dissemination. Poland also gave high scores for four of the activities but had lower scores for Prioritisation and Assessment. Bulgaria’s highest scores were for Identification and Assessment.

For Policy makers there were some areas of agreement between countries but also differences. For Kazakhstan the highest score was for Assessment, followed by Prioritisation and Producing information. Poland also had high scores for Assessment and Producing information but the highest score was for Internal review. Bulgaria’s highest score was for Prioritisation followed by Assessment and Reviews.

Both Kazakhstan and Poland had relatively high scores for Product makers in Diffusion and dissemination. Kazakhstan had a high score for Producing information which was not matched by Poland. Bulgaria’s scores were relatively low for all activities.

For Principal investigators, Kazakhstan and Poland both had high scores for Prioritisation and Assessment. Kazakhstan also had a high score for External review while Poland had a low score. Bulgaria had high scores for Assessment, Product information and Reviews.

With Public, the highest score was from Bulgaria for Identification. Kazakhstan and Poland had significant scores for Prioritisation, lower scores for other items.

Table 2 presents the choices of respondents from the three countries on six objectives for stakeholders’ participation in HTA.

Table 2 Level of Agreement on Objectives of Stakeholders’ Participation in HTA

Objectives of stakeholders’ participation in HTA is to enhance the completeness of the evidence underlying decision-making for personalized health technologies were agreed by representatives of the all three countries, though the Polish response included a high proportion of partial agreements.

Kazakhstan and Bulgaria were in agreement with legitimacy of the preventive technology assessment process, thereby ensuring public perception of the results. However, the Polish response, with a high proportion of partial agreements, indicated that almost 10% of participants disagreed and that support was not achieved.

In contrast, the task of stakeholders’ involvement in HTA is to increase the ability of the population to take care of their own health, all responses from Poland were in full or partial agreement, but the other countries showed that over 23% of their responses disagreed.

The objective as to improve public participation in the democratic process was supported by all the responses from Kazakhstan, but both Poland and Bulgaria indicated substantial levels of disagreement.

Objectives of stakeholders’ participation in HTA is to ensure that the HTA process is consistent with public values and the public interest is included in the values of the HTA process as well as Help explain the reasons for difficult decisions to the public attracted strong support from both Kazakhstan and Bulgaria but limited approval from Poland with almost 10% disagreement (Table 2).

Selection and Management of Stakeholder’s Participation in HTA Process (Open Question)

All respondents agreed that MoH should select stakeholders’ in HTA preparation and decision-making process. However, respondents from Kazakhstan and Bulgaria note that in addition to representatives of the MoH, it is necessary to input from the HTA body, patients and industry.

All respondents from the three countries agreed that regulatory (guidance) documents should be created to attract stakeholders. They would include selection of participants, their rights in the HTA process, and the stage of their involvement of HTA.

Representatives from industry of Kazakhstan noted that to select stakeholder’s an online application must be formed, then the Advisory commission (on reimbursement) should conduct a selection considering the criteria and competencies.

From patient perspectives, the independent body (non-government organization) with competencies and clear guidelines should provide management of stakeholders’ involvement in HTA preparation and decision-making process. Moreover, this independent body should be competent to identify conflicts of interest.


Democracy process in some middle or low middle income countries developed slowly. Implementation of the deliberative process as an aspect of the democracy can help to include public opinion with social values.16,17 The development of HTA in Kazakhstan5 has led to changes in the approach to health benefits package design in that country.9 That example would have informed Kazakhstan respondents’ opinion that stakeholders’ objective is public participation in the democratic process. In the context of Poland, a noteworthy segment of respondents conveyed a nuanced perspective, demonstrating partial alignment with the objectives outlined in HTA. This tendency could be ascribed to the intricate landscape of health policy within the country, which has been characterized by ongoing reform initiatives aimed at restructuring healthcare services. The presence of such reform proposals has likely introduced a level of uncertainty and ambiguity, which in turn may have influenced respondents to exhibit varying degrees of agreement with HTA objectives.18,19

There is no consensus between all respondents at what stage stakeholders should be involved in HTA. However, the work notes the involvement of patients or society at various stages, such as identifying health technologies for assessment, prioritization process, assessment or conducting evaluations, produce information and making recommendations, implementing funding recommendations, and disseminating solutions.20–22 According to respondents, for the Kazakhstan HTA model, patient or public involvement can be considered at all stages, except for the internal and external review of the HTA report. For Poland, their involvement is most likely should be at the stages of identification, prioritization, assessment and dissemination, while in Bulgaria identification, produce information, external review and dissemination.

It is worth noting that patient-oriented groups are not sufficiently developed in Kazakhstan. Often, some nongovernmental organizations of individual patient-centered groups may have goals related only to their activities, while other areas may not be sufficiently discussed. When patient-centered groups or individuals are involved, it is important that they contribute to the HTA process by considering the value of all patients as a whole, rather than specific groups with a particular disease.

In many stages of HTA assessment, respondents from Kazakhstan see an important role for the participation of purchasers, payers, and policy makers in almost all stages, where in the other two European countries their role is rated slightly lower. This may be due to the fact that in Kazakhstan all three stakeholders work closely together. The Social Health Insurance Fund forms a tariff at which a medical organization receives funds to reimburse services, therefore they are interested in introducing cost-effective technology. This may be why respondents proposed their participation in all HTA processes.

Responses for Bulgaria reflected its use of adopted European regulations as a member of the EU.23 Implementation of these is in progress, and will be reflected in regulatory documents.

There is no consensus between respondents on how stakeholders should be selected, or on the capacity and role of different types of stakeholders. This may be due to the fact that stakeholders are not sufficiently aware of the role of each participant and the HTA processes. For example, in the work of Scott and co-authors determined that patients could not always clearly imagine the purpose of their participation in HTA, although sometimes they could provide information about what contribution they made.20 In this regard, the authors of this work believe that it is necessary to provide training activities for stakeholders on basic issues of the HTA process, so that each stakeholder can make their contribution in decision making process.


In all three countries, Bulgaria, Kazakhstan, and Poland, progress is observed in relation to stakeholders’ involvement in HTA. The fact that these countries have started discussions and they are ready to apply the best practices in their countries shows that the health system is ready for change. Future steps for stakeholder engagement can start from development of regulatory (guidance) documents that will include the selection of stakeholders and the management of their involvement or participation.

Data Sharing Statement

Materials and these articles are with the authors.

Ethical Approval

The study was approved by the Local Ethics Committee of KazNMU (№8 (114) from 30.06.2021), Almaty, Kazakhstan.

Consent for Publication

All authors consent for publication.


Article submission funding provided by Medical University Sofia (Bulgaria).


The authors declare they have no competing interests in this work.


1. O’Rourke B, Oortwijn W, Schuller T.; International Joint Task Group. The new definition of health technology assessment: a milestone in international collaboration. Int J Technol Assess Health Care. 2020;36(3):187–190. doi:10.1017/S0266462320000215

2. van der Wilt GJ, Bloemen B, Grin J, et al. Integrating empirical analysis and normative inquiry in health technology assessment: the values in doing assessments of health technologies approach. Int J Technol Assess Health Care. 2022;38(1):e52. doi:10.1017/S0266462321001768

3. Baltussen R, Paul Maria Jansen M, Bijlmakers L, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256–260. doi:10.1016/j.jval.2016.11.019

4. Yazdizadeh B, Shahmoradi S, Majdzadeh R, et al. Stakeholder involvement in Health technology assessment at National level: a study from Iran. Int J Technol Assess Health Care. 2016;32(3):181–189. doi:10.1017/S0266462316000167

5. Kosherbayeva L, Hailey D, Kurakbaev K, et al. A process of prioritizing topics for health technology assessment in Kazakhstan. Int J Technol Assess Health Care. 2016;32(3):147–151. doi:10.1017/S0266462316000222

6. Decree of the minister of health on the agency for health technology assessment in Poland [Zarządzenie Ministra Zdrowia zmieniające zarządzenie w sprawie Agencji Oceny Technologii Medycznych]. Available from: Accessed December 18, 2023.

7. Ordinance on the terms, rules and procedure for regulation and registration of prices for medicinal products, Art. 1, item 8 and Chapter Six (SG, issue 26 of 2019, in force from 01.04.2019). Available from: Accessed December 18, 2023.

8. On approval of the rules for assessing health technologies and their application order of the minister of health of the Republic of Kazakhstan dated November 30, 2020 No. KR DSM-215/2020. Registered with the Ministry of Justice of the Republic of Kazakhstan on December 2, 2020. Russian and Kazakh. Accessed May 14, 2024.

9. Oortwijn W, Surgey G, Novakovic T, Baltussen R, Kosherbayeva L. The use of evidence-informed deliberative processes for health benefit package design in Kazakhstan. Int J Environ Res Public Health. 2022;19(18):11412. doi:10.3390/ijerph191811412

10. Medicinal products in human medicine, Art. 259, para 1, item 6 of (SG, issue 102 of 2018, in force from 01.01.2019); Bulgaria regulatory document.

11. Pichon-Riviere A, Soto N, Augustovski F, Sampietro-Colom L. Stakeholder involvement in the health technology assessment process in Latin America. Int J Technol Assess Health Care. 2018;34(3):248–253. doi:10.1017/S0266462318000302

12. Cavazza M, Jommi C. Stakeholders involvement by HTA Organisations: why is so different? Health Policy. 2012;105(2–3):236–245. doi:10.1016/j.healthpol.2012.01.012

13. Concannon TW, Meissner P, Grunbaum JA, et al. A new taxonomy for stakeholder engagement in patient centered outcomes research. J Gen Intern Med. 2012;27(8):985–991. doi:10.1007/s11606-012-2037-1

14. Street J, Stafinski T, Lopes E, Menon D. Defining the role of the public in Health Technology Assessment (HTA) and HTA-informed decision-making processes. Int J Technol Assess Health Care. 2020;36(2):87–95. doi:10.1017/S0266462320000094

15. Patient Involvement in Health Technology Assessment in Europe. Results of the European Patients’ forum Survey. Available from: Accessed May 14, 2024.

16. Goetghebeur M, Cellier M. Deliberative processes by health technology assessment agencies: a reflection on legitimacy, values and patient and public involvement comment on “use of evidence-informed deliberative processes by health technology assessment agencies around the globe”. Int J Health Policy Manag. 2021;10(4):228–231. doi:10.34172/ijhpm.2020.46

17. Syrett K. Expanded HTA, Legitimacy and Independence Comment on Expanded HTA: enhancing Fairness and Legitimacy. Int J Health Policy Manage. 2016;5(9):565–567. doi:10.15171/ijhpm.2016.75

18. Sagan A, Kowalska-Bobko I, Badora-Musiał K, Gałązka-Sobotka M. A reform proposal from 2019 aims to improve coordination of health services in Poland by strengthening the role of the counties. Health Policy. 2022;126(9):837–843. doi:10.1016/j.healthpol.2022.06.006

19. Sowada C, Sagan A, Kowalska-Bobko I, et al. Poland: health system review. Health Syst Transit. 2019;21(1):1–235.

20. Scott AM, Wale JL; & On behalf of the HTAi Patient and Citizen Involvement in HTA Interest Group, Patient Involvement and Education Working Group. Patient advocate perspectives on involvement in HTA: an international snapshot. Res Involv Engagem. 2017;3(2). doi:10.1186/s40900-016-0052-9

21. Menon D, Stafinski T. Role of patient and public participation in health technology assessment and coverage decisions. Expert Rev Pharmacoecon Outcomes Res. 2011;11(1):75–89. doi:10.1586/erp.10.82

22. Gagnon MP, Desmartis M, Lepage-Savary D, et al. Introducing patients’ and the public’s perspectives to health technology assessment: a systematic review of international experiences. Int J Technol Assess Health Care. 2011;27(1):31–42. doi:10.1017/S0266462310001315

23. Regulation (EU) 2021/2282 of the European parliament and of the council of 15 December 2021 on health technology assessment and amending directive; 2011. Available from: Accessed May 14, 2024.

Creative Commons License © 2024 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.