Validation of a Scorecard of Quality and Activity Indicators for Telepharmacy Pharmaceutical Care Services in Spanish Hospitals

Introduction

Telepharmacy, the remote pharmaceutical activity performed through information and communication technologies (ICT),^1,2 is a healthcare modality carried out in community care structures,³ primary care^4,5 and hospital care^6–12 and is included within the scope of telemedicine and telehealth.¹³ It has demonstrated numerous advantages in healthcare, both from the clinical and economic point of view and in terms of perceived quality.^14–17 Telepharmacy is not limited to remote dispensing through ICT, but encompasses a wide range of activities ranging from the relationship with the professionals, training and information for patients and professionals, pharmaceutical validation, or pharmacotherapeutic monitoring, as has been stated by various scientific societies.^1,2 Pharmaceutical care through Telepharmacy has experienced an exponential increase worldwide following the covid-19 pandemic, while preserving the health of pharmacists and patients.^18–20

Telepharmacy must be monitored within a quality management system to guarantee efficiency, safety, and quality of the activities it encompasses. At the hospital level, and as part of MAPEX²¹ (acronym in Spanish for Strategic Map of Pharmaceutical Care for hospital outpatients), the SEFH (acronym in Spanish of Spanish Society of Hospital Pharmacy) has presented its positioning and strategic framework on Telepharmacy¹ and has proposed the first scorecard of quality and activity indicators for Telepharmacy (TIS) worldwide (Appendix I).²² This TIS defines 50 indicators (31 priority), selected by consensus techniques for their appropriateness, usefulness/relevance, and feasibility. Indicators were grouped into 5 areas of application of telepharmacy: general and cross-cutting aspects; pharmacotherapeutic follow- up; remote dispensing and informed drug delivery; patient training and information; and coordination with the health care team. In addition, a glossary of terms and a computer tool for calculating the indicators were published (Appendix II). However, this scorecard has not been validated for global application by hospital pharmacy services, which is a major limitation for the extension of its application in healthcare. In addition, to our knowledge, this type of scorecard does not exist in other countries, neither published in scientific literature nor available from various professional pharmaceutical or medical associations. Furthermore, considering the different positions on telepharmacy from other organizations and associated pharmaceutical activities (such as the American Society of Health-care Pharmacist), we believe that many of the indicators validated in this study are directly or indirectly applicable to other hospital pharmacy services in many countries.

The objective of this project is to validate this TIS for its implementation in hospital pharmacy services. As a secondary objective, to know the degree of implementation of Telepharmacy in our country.

Materials and Methods

A working group with 7 hospital pharmacists who are national experts in Telepharmacy was formed. This group had been part of the working groups for the elaboration of the methodological support documents of the SEFH Telepharmacy Strategic Framework, among them the TIS. The project was developed in the following phases.

Phase 1. Elaboration of the Validation Questionnaire. Validation Criteria

To assess the suitability of the TIS and its supporting tools and contents and to measure and evaluate the implementation and development of the different areas of Telepharmacy in the Hospital Pharmacy Services (HPS) of different centers of varying complexity throughout the country, a questionnaire to determine the following five characteristics of the MAPEX-SEFH TIS was developed:

1. If it is holistic: if it integrates Telepharmacy from all perspectives. 6 questions related to indicators 1,9,14,19,24,29.
2. If it is practical: if it has practical utility for the different pharmacy services. 6 questions related to indicators 5,13,18,23,28,33.
3. If quantitative: if it includes quantitative indicators for measurement. 6 questions related to indicators 8,12,17,22,27,32.
4. Its usability: the ease with which a user can use the scorecard. 9 questions related to indicators 2,3,4,6,11,16,21,26,31.
5. If it promotes continuous improvement. 6 questions related to indicators 7,10,15,20,25,30.

The questionnaire, after 4 rounds of consensus, was structured according to the 4 areas covered by the TIS, preceded by a block of general questions on the TIS and the support tool. For each question, each hospital pharmacy service was asked to select its degree of agreement or disagreement on a scale of 1 to 9, with 1 being “strongly disagree” and 9 “strongly agree”. In addition, in each of the sections, there was a free text option for additional comments. The validation questionnaire is included in Appendix III.

The validation criteria for each of the 5 TIS characteristics was performed using the RAND/UCLA methodology, an adaptation of the Delphi methodology, and the following criteria were followed:^23–25 the characteristic was considered validated when the median was found to be within the score range 5–9 and at least 2/3 (66.66%) of the respondents the scored in the range containing the median. Conversely, the characteristic was considered not validated when the median was in the range 1–4 or more than 1/3 (33.33%) of the respondents scored it outside the range that contained the median.

Phase 2. Selection of Hospitals in Which the Study Was Carried Out

The sample of hospitals in which the survey was conducted was selected based on the number of hospital beds, in the set of all Spanish hospitals with hospital pharmacy services and in all regions of the country (200 hospitals). The population and sample included hospitals regardless of whether they carried out telepharmacy activities or not. For a confidence level of 95% and a margin of error of 15%, a minimum sample of 41 hospitals was estimated. Considering a 20% loss rate, 54 hospitals were selected in a stratified way: 4 level 1 (<100 beds); 17 level 2 (101–250 beds), 11 level 3 (251–500 beds); and 22 level 4 (>500 beds).

Once the hospitals had been selected, those responsible for the telepharmacy program of the pharmacy service and/or the heads of service were contacted to request participation in the validation project. Respondents who agreed to participate received telephone information about the TIS as well as the objectives of the study, methodology, and how to complete the questionnaire. For a correct response to the survey, the following information was sent by email: invitation letter, TIS, indicator calculation tool, and validation questionnaire.

Phase 3. Completion of the Validation Questionnaire by the Selected Hospitals

To complete the validation questionnaire, an online form was designed, where managers could enter general data and their ratings for each of the items consecutively. A QR code was also provided as an alternative to the online form so that the questionnaire could be completed using a mobile device. All managers from the hospitals that participated in this study provided their specific consent for participation and for completing the questionnaire.

A period of 2 months was allowed for this phase, during which the participants were able to interact with the members of the working group to resolve any doubts related to the completion of the questionnaire.

Phase 4. Analysis of the Results, Proposal of Conclusions and Preparation of the Final Document

Survey results were aggregated and analyzed qualitatively and statistically. Statistical analysis of the responses included the following metrics: median, interquartile range (IQR) and arithmetic mean. The Mann–Whitney U-test was used to assess whether the difference between the values obtained was statistically significant. The statistical significance value of was p < 0.05. The software used for analysis was Microsoft Excel v.2021.

The consultant produced a report with these results, with the comments or observations of the respondents and with initial conclusions. All of them were analyzed in a joint meeting with the working group of specialists in Telepharmacy, which agreed to draft a final document that includes the results, the final conclusions, and a selection of 14 essential indicators that would be accessible to all hospital pharmacy services and that would allow the evaluation of performance in the field of Telepharmacy and continuous improvement.

Following Spanish legislation this study does not require prior approval from a Research Ethics Committee due to the following concurrent circumstances: absence of patient data; methodological and non-interventional nature; no involvement of medications or clinical procedures; compliance with data protection regulations.

For the development of the project, we used the services of an external consulting firm (Ascendo Sanidad&Farma) with national experience in the management of technical health documents, healthcare validation methodology, experience of collaboration with the SEFH, knowledge of the activity carried out by hospital pharmacy services and who had previously collaborated in the definition of the TIS.

Results

Of the 54 hospitals selected, 44 centers from all regions of Spain and the 4 levels of complexity according to the number of beds responded, so that the centers participating in the study can be considered representative of the national population of hospital pharmacy services (Figure 1).

Figure 1 Hospital centers participating in the study, by region and level of care.

Eighty-eight percent (39/44) of the centers participating in the pilot study carried out Telepharmacy activities, with the Canary Islands and La Rioja being the communities in which no Telepharmacy activities were detected among the participating centers. The development of telepharmacy based on its areas of application are shown in Figure 2. Of the centers that do carry out telepharmacy activities, 43.5% (17/39) do not measure indicators of any type (activity, quality, efficiency, etc).

Figure 2 Development of telepharmacy activities in Spanish hospitals based on its areas of application.

The results of the questionnaire for validation²⁶ of the TIS characteristics were as follows:

● Holistic: 8.2. 98.5% of responses >5.

● Practical: 7.9. 98.9% of responses >5.

● Quantitative: 7.9. 98.6% of responses >5.

● Usability: 6.9. 87.37% of responses >5.

● Continuous improvement: 7.9. 100% of responses >5.

The median scores for each of the questions, as well as the percentage of responses whose median was between 7 and 9 for each of the characteristics required of the TIS, can be seen in Table 1. No significant differences were observed in the results depending on whether or not the participating center carried out Telepharmacy activities, except concerning the usability of the TIS, 6.9 vs 7.9; p=0.0059 (Figure 3).

Table 1 Median Scores for Each Question in the Questionnaire and Percentage of Them Between 7 and 9

Figure 3 Median scores obtained for each of the TIS characteristics evaluated.

In addition to the quantitative results, 4 comments or observations have been expressed by at least 10% of the respondents: difficulty in obtaining data for the measurement of the indicators (47.7%), mainly due to the lack of integration and/or automation for data collection and extraction (34.1%); the TIS is comprehensive, covering a wide range of indicators relevant to Telepharmacy, and very exhaustive on some occasions, as it requires an excessive level of detail in some aspects (18.2%); it is difficult to obtain cost data (13.6%). Likewise, there are 10 comments regarding the lack of indicators related to Telepharmacy, which were, however, included in the TIS.

Based on the analysis of the results of the validation questionnaire and the comments of the respondents (especially concerning usability), the authors selected by unanimous consensus 14 essential indicators, stratified as basic or advanced, that would allow all Spanish pharmacy services to know the implementation and improve the quality of pharmaceutical care through Telepharmacy (Table 2) (formulation and descriptive sheets are available at https://www.sefh.es/mapex/cmo-oportunidad.php).

Table 2 Selected Essential TIS Indicators, Stratified as Basic or Advanced

Discussion

This study has validated a TIS for use in hospital pharmacy services proposed by the SEFH. To this end, a national survey was carried out in a valid sample of hospitals at the four levels of care in all regions of Spain. To our knowledge, this is the first TIS validated for use in the healthcare setting. Furthermore, it will be very useful at the hospital level, since practically all the hospitals surveyed use Telepharmacy in its different areas of application, but almost half of them do not use indicators of any kind to monitor this activity.

Modern healthcare organizations, such as hospitals, handle a large amount of clinical, administrative, and economic information which, as part of an efficient management system, must be quantified, evaluated, and monitored to identify performance and trends, establish corrective measures and improve the quality of care.²⁷ Indicator scorecards are the management tools that make it possible to carry out these interventions since they include indices for monitoring the organization as a whole, specific areas or departments, groups of patients with the same pathology, or a care procedure.²⁸ On the other hand, in the hospital setting, validation of any healthcare technology, care procedure, or proposed management tool (such as scorecards) is very important, as it provides the internal and external validity necessary for its correct use within and outside the organization.²⁹ In this sense, Telepharmacy, as a growing healthcare procedure, should also be monitored and evaluated through a scorecard of indicators such as the one developed by the SEFH²² and validated for use in the hospital setting.

The validation methodology used is based on the “virtual design and construction (VDC)” method of Stanford University³⁰ and has already been widely used for the design and validation of scorecards;³¹ it is a methodology based on the use of multidisciplinary performance models integrated in project design to support explicit and public objectives of a scorecard of indicators and integrating an evaluation of four characteristics: adaptive, quantifiable, holistic and practical. In our study, these four characteristics were evaluated, to which a fifth was added, namely, whether it favored continuous improvement; this is justified on the basis that the proposed TIS not only includes quantitative activity indicators but also introduces a significant number of indicators that allow the monitoring of the quality of care in Telepharmacy.

The results obtained for all the characteristics evaluated show a broad consensus on the validity of the proposed TIS. Of the 5 characteristics, 4 of them (holistic, practical, quantitative and continuous quality improvement) obtained a median rating of around 8 out of 9, with around 85% of the ratings above 7. The study confirmed the holistic property of the TIS, a central characteristic of a TIS;^32–35 this means that it applies to all the proposed procedures or applications of Telepharmacy (pharmacotherapeutic monitoring, informed dispensing and delivery, training and information of patients and professionals, and coordination with the care team) and that it includes all the dimensions of the use of Telepharmacy (organization, regulation and ethics, care activity, patient and professional experience, training, human resources, economic evaluation, quality, clinical effectiveness, and logistics).

Likewise, its practical usefulness, a fundamental characteristic for its use,^36–39 has been widely validated; its practicality is also supported by several documents that complement the TIS, such as the descriptive sheets for each indicator (detailing aspects to facilitate its evaluation, such as: justification or quality criteria, dimension, formula, priority, explanation of terms, population, type, periodicity, standard, data sources and comments), the repository of variables and the glossary of terms (to facilitate the collection of the data): justification or quality criteria, dimension, formula, priority, explanation of terms, population, type, periodicity, standard, data sources and comments), the repository of variables and the glossary of terms (to facilitate the collection of the quantitative variables to be evaluated in the Scorecard and the understanding of the terms included in the fact sheets), the work tool (which facilitates the collection of the variables to be measured to evaluate the implementation of the indicators) and or general recommendations for the proper use of the TIS (which informs about specific considerations on the measurement of the indicators).

In addition, the inclusion of quantitative indicators has been very positively valued, although they have had limited use in the healthcare field and very few have been validated beyond the opinion of experts.⁴⁰ The TIS evaluated in our study includes a minority of qualitative indicators corresponding to general strategic aspects of the implementation of Telepharmacy (general work procedures, quality plan, training plans, contingency plan, etc.) and most of the indicators are quantitative and cover all areas of application of Telepharmacy. This will allow not only a temporal monitoring of the intra-hospital performance, but also a comparison with other hospital centers.

On the other hand, the respondents considered that the TIS promotes the improvement of the quality of care. This is due to the inclusion of: general qualitative indicators related to the need to have a Strategic Plan in the HPS (with objectives, lines of action, organization, etc., for the performance of Telepharmacy); quantitative indicators by area of application; and the definition of quality standards aligned with the strategies of the European Observatory on Health Systems and Policies.⁴¹ In addition, it is very important to note that the already validated TIS itself and its impact on continuous quality improvement can facilitate a cycle of feedback and constant improvement in the practice of Telepharmacy, in the environment of the QPEX Standard for certification of the quality of pharmaceutical outpatient care of HPSs.⁴²

The fifth characteristic that has been evaluated for the validation of the TIS has been usability. It obtained a rating of around a median of 7, with around 60% of surveys with an evaluation of more than 7 out of 9 (a rating slightly lower than that of the other 4 characteristics mentioned above). In addition, respondents have made numerous comments or observations regarding this characteristic concerning the difficulty of obtaining the data necessary to measure the indicators. This is not a minor issue, given that it is very difficult to monitor quality or activity indicators without efficient access to the necessary data. Various limitations related to the interoperability of hospital information systems, the difficulty of accessing certain data due to the demanding conditions related to cybersecurity or the scarce digitization of data in the healthcare field justify the assessment given to usability, as has also been reflected in recent publications.^28,43–45 Therefore, for the TIS to be successfully implemented in the HPSs, it is necessary to face certain challenges related to the availability of the data needed to obtain the indices defined for each of the indicators that make up the Telepharmacy mantle chart. This involves, among other actions, the inclusion and measurement of the activities carried out in the HPS in the hospital information systems or the digitalization of the information.

The importance of using this validated telepharmacy dashboard is twofold. On the one hand, it establishes quality criteria for this type of pharmaceutical care. On the other, it allows for the use of validated indicators to compare the activity and quality of this procedure, not only to understand in-hospital outcomes but also to compare activity and quality across different hospitals. In both cases, it is a valid tool for continuous improvement in the quality of care.

The validated TIS can constitute a relevant management tool in the field of hospital telepharmacy in our country. But, could it be used in other healthcare settings or could its use be generalized, totally or partially, in other countries or healthcare systems? This will basically depend on the concept and scope of Telepharmacy as a complementary activity in other healthcare contexts. In this sense, considering the positions of pharmaceutical scientific societies or the experiences developed throughout the world,^12,13,17,46 we consider that many of the indicators of this now-validated scorecard can be used in the management of telepharmacy in a large number of countries. However, on one of the limitations could be derived from the generalizability of any indicator scorecard since the application of telepharmacy might be different in each country and cultural and economic disparities may limit the use of such a tool. Therefore, similar studies in different countries are needed to ensure its validity and this should also inspire founders to propose customization of the tool to fit the conditions in different countries. Also, the main challenge that may arise from using this scorecard in other countries may arise from the different levels of implementation of telepharmacy and the accessibility of data needed to calculate the indices required for quantitative quality or activity indicators. For those hospitals that, for various reasons, cannot implement all the proposed indicators, this scorecard can serve as a guide to quality criteria for implementing different telepharmacy activities, using qualitative indicators classified as general. Once the general methodological foundations have been established, the quantitative indicators proposed as essential and basic indicators for this study can be used.

Future lines of research related to the Telepharmacy TIS, once validated, will be to determine its degree of implementation in hospital pharmacy services, which will allow not only to monitor this activity temporarily in each of the centers, but also to compare its development between different hospitals at different levels by having the measurement of valid indicators calculated in a homogeneous way.

Conclusion

In conclusion, based on the results obtained and the above discussion, this scorecard of quality and activity indicators for telepharmacy pharmaceutical care services, as well as the supporting documents and tools have been validated for Spanish hospitals. Hospital information systems are needed to allow efficient extraction of the data necessary to obtain the indices for each of the indicators and to improve the usability of the TIS.