The Effectiveness of the Joint Commission International Accreditation in Improving Quality at King Fahd University Hospital, Saudi Arabia: A Mixed Methods Approach

Introduction

The global interest in improving health care quality has caused many decision-makers to adopt standardized processes to evaluate health organizations. Pursuing the Joint Commission International Accreditation (JCIA) is one of the most popular tools adopted to achieve these goals. The accreditation process is usually a voluntary external evaluation of a health care organization by measuring its compliance level against a set of predetermined standards. These standards provide a basis for a comprehensive assessment of a health organization’s performance. Based on the evaluation results, policies and interventions are implemented to improve these organizations’ performance.¹ Over 1000 health organizations worldwide pursued the JCIA’s since it is viewed as the world’s leader in health care accreditation. Furthermore, the JCIA’s golden seal of approval is considered by administrators and stakeholders as an indicator of a hospital’s high quality of services and a guarantee of patient safety.² This trend is also apparent in Saudi Arabia, which ranks second in the list of countries with the highest number of accredited health organizations.³

Despite the popularity of the JCIA, its effectiveness in improving the quality of health services remains controversial.⁴ This controversy is due to the inconsistent evidence on the impact of accreditation on health services, patient outcomes, and organizational cultures.⁵ This lack of consensus is due to the difficulty in measuring the outcomes of this dynamic and complex intervention.⁵

The majority of published literature focused on assessing the impact of the JCIA on quality outcomes. For instance, Devkaran & O’Farrell utilized an interrupted time series analysis to investigate the impact of the JCIA on hospital quality measures. The research was conducted in a 150-bed multi-specialty hospital in Abu Dhabi during a 48-month period. Moreover, the differences in 27 quality performance measures, such as mortality rates, were compared between two time periods: one-year pre-accreditation (2009) and three years post accreditation (2010, 2011, and 2012). According to the results, the formal JCI survey preparation had a significant positive impact on 74% of the quality measures. However, accreditation had a negative impact on 48% of measures and a positive effect on merely 4% of measures during the post-accreditation period. Nevertheless, the study found a residual benefit three years after the intervention and found that the improvements were maintained at 90%, which is 20% greater than the baseline level in 2009. The researchers concluded that despite the drop in performance after the survey, the results showed that the improvements from the accreditation were sustained during the three-year accreditation cycle.⁶

Several studies investigated the perceptions of health care providers towards this process and the factors affecting it. According to Despotou, Her & Arvanitis, nurses working in a university hospital in Korea had an overarching positive perception towards international accreditation due to its positive impact on quality outcomes and patient safety. Furthermore, participants attributed the positive changes in the health organization to pursuing the JCIA. Nevertheless, nurses also highlighted some drawbacks of this process, such as the high cost and workload increase.⁷

Other studies explored the perceptions of various providers such as physicians, radiologists, medical technologists, and other allied healthcare professionals. Algahtani et al conducted a cross-sectional to study the perceptions of health professionals towards this process. The research was conducted in a Saudi university hospital, utilizing a questionnaire that measured the provider’s degree of participation in accreditation, the benefits of accreditation, and the quality of accreditation results. The overall results indicated that the JCIA had a positive impact on the process and health services provided at the hospital; nevertheless, the degree of the impact varied between the different health providers.⁸ Additionally, a systematic literature review conducted by Agustine & Pujiyanto concluded that the majority of studies reported that various health professionals perceived the effect of accreditation as positive in improving health services.⁹

Despite the abundance of literature on this topic, no previous studies aimed to assess the impact of accreditation on quality outcomes while providing context. To accomplish this goal, this research was the first to utilize a Convergent parallel mixed method approach to assessing the effectiveness of the JCIA in improving quality outcomes while investigating perceptions of health professionals and employees at KFHU.¹⁰

Materials and Methods

Mixed Methods Framework

A Convergent parallel mixed methods approach was used to meet the two main aims of the study. The study’s first aim was accomplished utilizing an interrupted time series analysis to assess the impact of the JCIA on a total of 12 outcomes. [Table 1] The second aim was to investigate the attitudes and perceptions of health professionals using a qualitative approach to identify key challenges in implementing the JCIA.⁸ Using this framework will more likely yield useful results since the impact of a complex intervention, such as accreditation, is difficult to interpret using either method independently. Moreover, pursuing accreditation can cause changes that cannot be assessed using quantitative measures such as the changes in the behavior and health providers’ attitudes. Additionally, qualitative data can complement quantitative data by providing context that might explain the possible causes of the changes or lack thereof in some quality outcomes. Therefore, a Convergent parallel design was selected for this research since it prioritizes both types of data.¹¹

Table 1 Description of Each KPI

Based on this framework, the first step was to collect and analyze quantitative and qualitative data separately. This step included developing the research questions, identifying the study sample, and the methodology for both quantitative and qualitative data. The quantitative data were collected using KPIs that measure each quality outcome to determine whether it improved before, during, and after the accreditation. The qualitative data were collected during individual interviews and focus groups with administrative employees and different health providers at King Fahd University Hospital.¹¹

The third step is to report the results of each section. The quantitative results were reported in both a table format and time series graphs to illustrate the patterns in the study outcomes before, during, and after the intervention period. As for the qualitative results, they were reported in tables followed by more descriptive analysis. Lastly, the mixed results were interpreted; this step is also referred to as the point of interface. During the last step, “the researcher identified relationships, contradictions, convergence, and divergence in the different sources of data.”¹¹

Study Population

King Fahd Hospital of the University (KFHU) is a public hospital that provided free health services to Saudi citizens and foreigners working in the public sector. The hospital is affiliated with Imam Abdulrahman Bin Faisal University (IAU), and its primary purpose is to train medical students during their clinical years. The hospital has a 550 bed-capacity to meet the increasing demands of the community. The total number of employees working in both clinical and nonclinical departments was 1361 employees (345 permanent and 1016 contractors), including nursing staff, quality analysts, health providers, dietitians, Information Technology (IT), and administrative staff. Furthermore, the total number of patients visiting the Emergency Room (ER) and outpatient clinics was about 463,498, and over 20,627 patients were admitted in 2017. The hospital’s administration initiated the preparations to pursue the JCIA in early 2014. After almost a year, the hospital was accredited for the first time in 2015. The JCIA is only valid for three years; therefore, to maintain the accreditation status KFHU imitated a second accreditation cycle in September 2018.

The quantitative data was collected from 12 KPIs that measure the changes in quality processes and outcomes. As for qualitative data, it was collected from 31 health providers through individual interviews or focus groups. The health providers fell into four categories, namely: 1) nurses, 2) physicians, 3) administrative staff, and 4) other types of providers such as radiologists and lab technicians. The researcher employed purposive sampling to ensure that the participants included health professionals from each category. The study subjects were identified either directly by the researcher, through the Directorate of Quality and Safety (DQS), the department heads, or other participants.⁹

Quantitative Analysis

One of the main issues with using comparative designs and cross-sectional studies in measuring the impact of the JCIA is that they can only establish correlations between variables. This issue is due to the accreditation process’s dynamic nature that has both short term and long-term outcomes. For that reason, an interrupted time series analysis (ITS) was selected to measure the impact of accreditation by dividing the regression into multiple posts and pre-intervention equal time intervals.^12,13 Moreover, it is also important to note that KFHU did not undergo drastic organizational structure changes or other significant interventions to improve quality during the study period, limiting potential bias in the results.

The DQS at the hospital collected the data that was used in this analysis. The data were collected retrospectively through a closed medical record review, observations, hospital information system, reports, and other documents. The Medical Records Department prepares 10% of randomly selected discharged patient records each month to be reviewed by the quality department. The KPIs were specific to the hospital and were selected by the DQS with some guidance from the JCIA consultants and were not adapted to the local context. The DQS’s employees then measured the KPIs to monitor compliance with the accreditation standards to track improvements in the quality of services. Data were available on excel sheets from the DQS and then were merged into one sheet and imported into the statistical software STATA. The data was clean due to going through secondary data analysis, and the researcher omitted only duplicate observations.

This study included data from the beginning accreditation preparation phase in January 2014 to data collected during the second accreditation period in September 2018. In other words, the study period included a full accreditation cycle and the beginning of the second accreditation cycle three years post the first time the hospital was accredited.

The conceptual model utilized for this study was the Donabedian model. This model was appropriate since the accreditation process is a compressive intervention that changes all components of the health care process, including inputs, processes, and outcomes. This model also provided a basis to develop measures for processes and outcomes.¹⁴

An interrupted times series segmented regression model was used. This approach enabled the researcher to estimate the constantly changing levels of compliance with JCIA standards while controlling for changes that might occur outside of the intervention.¹⁵ Furthermore, there is a total of 12 quality measures in this study [Table 1]. Data for each KPI were collected by the quality department every month. The study period was a total of 57 months for each quality measure included in this study. The frequency of data time points enabled the researcher to detect any changes in each outcome. The outcomes include percentages, proportions, and averages. Lastly, due to the large number of outcome measures, the Bonferroni correction was used to adjust for the p-value and to decrease the risk of an inflated type I error.¹⁶

The Following is the Interrupted Times Series Segmented Regression Model

Yt refers to the outcome, which was measured using a KPI for each measurable element of a JCI standard in this study. As for, t it is time in months at time_t starting from the start of the observation period on January 2014 to the last time point in the series on September 2018. Intervention refers to the formal survey conducted by JCI surveyors that resulted in officially accrediting the hospital in September 2015; It is a measure for time_t, and it is a binary variable which is recoded into 0= is for the time occurring before interventions and 1 after the intervention. Time after intervention is a continuous variable with the number of months post-intervention at time_t. β0 is the baseline level of the outcome at the start of the time series. β1 is the slope prior to accreditation (i.e., the baseline trend). β2 is the level change after the intervention. β3 is the difference in the slope from pre to post accreditation. Lastly, e stands for the random error term.

The regression model was used to estimate the trend and the level of the dependent variables before and after the intervention. The independent variable is time, which is interrupted by the accreditation survey. The results were analyzed by interpreting the level change directly after the intervention (B2) and the change in slope after the intervention (B3). The changes in the level and slope were used to estimate the increase or decrease in each quality outcome. The results for each KPI were reported individually and categorized into process and outcome measures.

Due to the limited studies that explored the impact of accreditation on the quality of services in Saudi Hospitals, this study served as an exploratory study. A range of KPIs were selected to prioritize which quality outcomes are the most impacted by this process as a basis for future research. Additionally, the greater number of quantitative outcomes allows for more opportunities to identify relationships and contradictions, with the qualitative data from the health providers’ perspectives. The total number of KPIs collected by the hospital was 113; nevertheless, only 12 of those measures were selected for this study.

The reason these measures were selected was based on exclusion criteria. The first criterion led to the exclusion of measures that were not directly related to the quality of health services or patient safety, such as facility management and laundry services associated measures. The second exclusion criterion included measures that pertained to a specific specialty or department, such as pediatrics and cardiology, to ensure that the outcomes applied to most patients treated at the hospital. The third criterion was KPIs that were not recorded throughout the period of study.

Furthermore, any indicators that were not collected every month were also excluded, such as the percentage of employees aware of the hospital’s safety management plan, which was only measured quarterly. Lastly, KPIs that had issues with their face validity were not included. For instance, if the source of the data collected for the measure was not verifiable or subject to bias were excluded.

Qualitative Analysis and Data Sources

The potential participants were identified either directly by the researcher, the quality department, the department heads, or by other participants. The first step of the participant recruitment process was compiling a list for each type of provider included in this study and contacting them in multiple ways. The first approach was directly emailing or calling the potential participants to set up an appointment to meet. If the response rate was low, the participants were contacted through a quality department employee or with an official letter from the department to request the participants’ cooperation in this study. Depending on the interviewee’s request or time restrictions, they were assigned to a focus group or an individual interview. No incentives were provided to participants to partake in either focus groups or interviews. The participants included a total of 31 participants, including physicians, nurses, administrative staff, and other health professionals. A breakdown of each participant category can be seen in Table 2.

Table 2 Qualitative Research Participant Profile

The participants were asked what their perceptions towards the changes resulting from pursuing the JCIA. This question prompted participants to discuss the positive and negative aspects of the accreditation process. Since this study was a part of a dissertation, the moderator’s guide and participant consent form were developed with the oversight of research supervisors before data collection. This oversight reduced the likelihood of researcher bias and enabled the researcher to develop neutrally framed questions and did not include any probing of participants.

Furthermore, data source triangulation was employed by collecting data from different health professionals’ categories to validate the accuracy of the data and gain multiple perspectives on the accreditation process. Both semi-structured focus groups and individual interviews were conducted. The focus groups were divided by type of provider to ensure that participants can provide their honest opinions. Subsequently, the researcher conducted individual interviews due to the interviewee’s request or to allow for more in-depth follow up questions.¹⁷

Thematic qualitative analysis was conducted solely by the researcher using the software NVivo. More specifically, the researcher utilized a hybrid approach of inductive and deductive coding and theme development. During the initial stage of the analysis, the researcher reviewed the transcripts from the interviews and focus groups to be familiar with the raw data. After that, the investigator utilized deductive coding using two open codes that mirror to categorize health providers’ perceptions. Subsequently, sub themes were identified from the data using inductive analysis, and axial codes were generated and sorted within each overarching open code. The following stage was to refine theme descriptions. Lastly, the data were reviewed multiple times to ensure their relevance to the study and to add additional codes when necessary.¹⁸

esults

Quantitative Results

As seen in Table 3, only six out of 12 outcomes had a statistically significant pre accreditation slope (β1) when considering the Bonferroni critical value. The pre accreditation slope suggests that the accreditation survey preparations, which started in January 2014, lead to statistically significant improvements in the percentage of hand hygiene compliance, nosocomial infections, and the percentage of radiology reporting turnaround time outliers. On the other hand, during the accreditation preparation, there was a monthly increase in the mortality rate and pressure ulcer rate. Furthermore, there was almost no change in the post-accreditation slope when it came to the percentage of laboratory critical values reporting within 30 minutes, which already had a compliance rate of 99.37%.

Table 3 The Impact of Accreditation on Quality Outcomes at King Fahd University Hospital

As for the improvement in level (β2), only the average length of stay had a statistically significant improvement, which was decreased by an average of 2.44 days. This finding suggests that the intervention had an immediate impact on the average length of stay. Conversely, there was a statistically negative impact on the level of two outcomes: the nosocomial infections and the rate of patients leaving the ER without being seen. Furthermore, there were slight improvements in the level of the percentage of laboratory critical values reporting within 30 minutes and the percentage of the radiology turnaround time outliers.

The change in slope (β3), which measures the sustained change in the post-accreditation period, showed a statistically significant improvement every month in the percentage of hand hygiene compliance, pressure ulcer rate, and mortality rate in the three years following the accreditation survey (2016, 2017, 2018). The percentage of laboratory critical values reporting within 30 minutes showed no additional improvements. Nevertheless, this lack of improvement is due to the already high compliance of reporting lab critical values within 30 minutes.

Furthermore, the results indicate that lab reporting compliance was maintained at 100% post-accreditation compared to the baseline trend, which can be considered a positive residual impact of accreditation. On the other hand, there was a sustained negative impact on the rate of patients leaving the ER without being seen.

Figure 1 contains graphs that illustrate each outcome’s patterns to better describe the changes throughout the study period. As seen from the graphs, the impact of an intervention on ITS may have immediate, lagging, or a combination of effects. An immediate change is reflected in the shift in the level immediately after the intervention (β2), and a lagged improvement is observed by the gradual change in slope (β3). This variation in the effects can be caused by how fast an outcome responds to the intervention.¹³ Regardless of statistical significance, the graphs suggest that nine out of 12 outcomes were improved throughout the accreditation process. The outcomes that did not improve after the accreditation process included the rate of patients who left the ER without being seen, the percentage of OR cancelations, and the rate of patient falls, which had both immediate and lagged increases.

Figure 1 Continued.

Figure 1 Timeseries graphs of quality outcome measures at King Fahad University Hospital. A: Timeseries graph of hand hygiene compliance before and after the accreditation survey. B: Timeseries graph of the rate nosocomial infections before and after the accreditation survey. C: Timeseries graph of the percentage of correct patient identification before and after the accreditation survey. D: Timeseries graph of the percentage of radiology report turnaround time outlier before and after the accreditation survey. E: Timeseries graph of the percentage of laboratory critical values reporting before and after the accreditation survey. F: Timeseries graph of the pressure ulcer incidence rate before and after the accreditation survey. G: Timeseries graph of the percentage of operating room cancellations before and after the accreditation survey. H: Timeseries graph of the percentage of patients leaving the ER without being seen before and after the accreditation survey. I: Timeseries graph of the mortality rate before and after the accreditation survey. J: Timeseries graph of the rate of patient falls before and after the accreditation survey. K: Timeseries graph of the average length of stay before and after the accreditation survey. L: Timeseries graph of the percentage of occupied beds before and after the accreditation survey.

Qualitative Results

The Perceptions of Health Professionals Towards the Accreditation Process

All the participants interviewed had an overall positive perception of the JCIA process, which fell into eight sub themes. [Table 4] A variety of health providers and administrators believed that quality improvement training and education at the hospital improved. For instance, one quality improvement employee stated that the nurses’ knowledge about patients’ rights and safety improved drastically due to the JCIA process. The participant believed that the increased interest in quality improvement training resulted from a shift in the organizational culture due to pursuing international accreditation.

Table 4 The Perceived Positive Impact of the JCIA* Process

Another perceived benefit was that meeting the JCIA standards resulted in a reduced amount of paperwork. The reduced paperwork was as a result of the JCIA’s consultant’s recommendation to merge duplicate forms. Examples of merged forms mentioned by participants include consent forms condensed from six forms into a couple of forms, and the multidisciplinary notes created instead of separate progress notes for each type of provider.

Furthermore, some participants noticed some improvements in quality outcomes, such as the reduction of medication errors. However, a greater number of participants believed that the JCIA mostly improved processes, policies, and procedures and not outcomes. For instance, a pharmacist stated that each department had its own IV administration process, but to meet the accreditation standards, they standardized all KFHU’s departments’ process. Moreover, two nurses believed that the JCIA might not have improved quality entirely, but it streamlined the health quality improvement process and organized it, making it easier to follow. According to a participant, the organization of efforts was possible because the JCIA engaged the hospital’s leadership with employees from all levels of the hospital toward a common goal.

The Negative Impact of International Accreditation

Despite all the participants having an overall positive perception of the process, they also highlighted some of its drawbacks. [Table 5] A total of nine negative effects of the JCIA process emerged from the qualitative data analysis. The most mentioned negative impact associated with the JCIA was the increased workload. A total of seven participants from all categories believed that meeting the JCIA standards increased their workload, and they had to meet the standards in a short period. According to some participants, the sudden additional workload to become internationally accredited distracted them from their primary job, which is patient care. For instance, one nurse stated, “I think they are making us focus more on the paper than patient care.”

Table 5 The Perceived Negative Impact Towards the JCIA* Process

Nevertheless, one nurse stated that this increased pressure is most likely temporary and will lead to a more efficient way to use the health providers’ time in the long run. She gave an example of how the transfer from paper documentation to a fully electronic system was time-consuming; however, it could lead to faster and more adequate documentation in the future. Furthermore, seven participants reported that the additional workload led to some resistance from providers to participate in the pursuit of accreditation. Six participants stated that this resistance led the hospital’s leadership to mandate compliance to meet international standards, which further increased the pressure on health providers. For instance, a quality improvement employee stated that the hospital’s president directly asked physicians to complete a great number of incomplete medical records within a strict deadline. According to participants, this was an ongoing issue during both accreditation cycles; however, two nurses stated that they believe that this issue is expected to be resolved in upcoming accreditation cycles since providers will be familiar with the process and its benefits.

Another main issue was selecting KPIs; one employee stated that

They didn’t enough help from the JCIA to select the most appropriate KPIs. She stated that the JCIA merely gave them some hints on what direction to go to select them.

Additionally, a physician reported how data for KPIs are collected can produce bias, particularly with observation-based KPIs such as the percentage of hand hygiene compliance. In other words, collecting data from observations may predictably produce a Hawthorne Effect as a result of health professionals temporarily modify their behavior due to being aware that they are being observed.¹⁹

According to six participants, another issue was that the improvements resulting from pursuing the JCIA might not be sustainable in the long run. One nurse explained that she believed that the improvements would be temporary unless the hospital’s culture shifts from being motivated by pursuing accreditation to a culture that prioritizes continuous quality improvement.

Three participants also expressed concerns that the accreditation process might result in the spread of misconceptions among hospital staff due to the misinterpretation of data. A quality improvement employee provided an example of how

The increase of OVRs [Occurrence Variance Reports] can be misunderstood as an increase in sentinel events; however, it is actually a good sign because it means more people are reporting them and not that they occur more often.

Three responses suggest that another concern was the limited communication both from the hospital’s leadership and between employees at all levels. Some medical record department employees felt that it was difficult for them to communicate their needs to other departments. Lastly, two participants expressed concerns that improving processes and quality outcomes may not be realistic without improving structures.

Discussion

Overall, this study’s quantitative findings indicate that the impact of accreditation was positive for the majority of outcomes. This result coincides with Devkaran & O’Farrell’s interrupted time series analysis, which concluded that the accreditation process positively impacted most study outcomes. Five out of 12 of this study’s outcome measures correspond with the Devkaran & O’Farrell. The outcome measures that this study and the UAE study share included: mortality rates, rate of patient falls, percentage of hand hygiene compliance, hospital-acquired infections, and OR cancelations. This study’s results demonstrated a statistically significant decrease in mortality rates pre-accreditation and a statistically significant decrease in the post accreditation slope. Whereas the Devkaran & O’Farrell study had no significant coefficients for mortality rates, the researchers concluded that this was since the JCIA is more process and structure focused and would not impact outcomes measures.⁶

According to this study’s results, the rate of patient falls slightly decreased by 0.01 directly after the intention and had a 0.01 decrease in the post accreditation slope (P≥0.004). As for the Devkaran & O’Farrell study, its results show that the rate of patient falls has increased by 0.21 (P≥0.05) and had a statistically significant sustained decrease by −0.67 per month, contrary to this research’s findings where the improvement was minor and temporary. This comparison suggested that the UAE hospital might have had a better action plan to continuously reduce the risk of patient falls, which yielded long term improvements.

On the other hand, hand hygiene compliance at KFHU decreased by 3.09% after the intervention (P≥0.004); however, there was an improvement in compliance in the post accreditation slope by 0.18% per month. This finding is inconsistent with the Devkaran & O’Farrell study that observed an increase in the level immediately after the accreditation but a decrease in hand hygiene compliance in the post accreditation slop. Poor hand hygiene compliance is an important indicator since it is a major contributor to hospital-acquired infections.²⁰ The rate of hospital-acquired infections in both studies somewhat coincide. This study indicates that nosocomial infections increased immediately after the accreditation, but it had a sustained decrease after the intervention. Similarly, there was an increase in hospital-acquired infections, but there was also a smaller increase in the post accreditation slope. However, the researchers of the UAE study attributed this increase in infections to the implementation of an infection surveillance program after the survey, which leads to an increase in the reporting of nosocomial infections.⁶

The qualitative analysis concluded that the JCIA’s impact was also perceived by all participants as positive and that it led to the overall improvement of health services and patient safety, which was also observed by Algahtani et al and Agustine & Pujiyanto. This study also identified nonquantifiable improved processes such as improved leadership involvement in quality improvement efforts. This finding was identified as a perceived contributing factor in improving the process in most previous studies and not as a benefit of accreditation.^8,9

Nevertheless, the participants highlighted some of its drawbacks, including the increased workload and health providers’ resistance to participate in this process; both were most common amongst doctors, according to participants. This finding coincides with Algahtani et al, which reported that physicians had the lowest accreditation process participation. The causes of the resistance to participate and whether or not this issue specifically applies to newly accredited hospitals remains to be investigated.⁸

Additionally, the utilization of a mixed methods framework allowed the researcher to better interpret the context of quantitative results and to ensure their accuracy. According to the quantitative results, the JCIA had a positive impact on most of the study’s outcome measures. This finding was supported by the qualitative data that showed that all the participants believe that the effect of international accreditation is mostly positive. According to Algahtani et al, there is a potential association between improved outcomes and the health providers’ positive perceptions.⁸ Nevertheless, the participants had mixed perceptions of the magnitude and the sustainability of the improvements, which is reflected in the quantitative results. The quantitative results indicated that the extent and the longevity of the improvements caused by the JCIA vary among the ten improved outcomes. The positive changes were either immediate, lagged, or both.

The improvements were sustained after the intervention except for the average length of stay, which only improved immediately after the intervention. These quantitative results contradict some of the participants’ statements that suggested that the JCIA formal survey caused temporary improvements due to a possible Hawthorne Effect. Nevertheless, a Hawthorne Effect may have been produced in KPIs that were based on observations at any time during the JCIA process. For instance, one physician stated that she did not perceive any drastic improvements in hand hygiene compliance from doctors after the accreditation, contradictory to what the quantitative results suggest. Nevertheless, the physician’s statement may not necessarily mean that hand hygiene did not improve, but it calls into question how observation-based KPIs were collected. The doctor’s comment suggests that there should be methods to ensure that proper hand hygiene is always followed by conducting continuous internal audits rather than occasional observations.

An additional issue with selecting KPIs is their focus on measuring two components of the health care delivery process, namely process and outcomes. Several participants shed some light on how certain improvements to meet the JCIA standard were difficult due to the limited staff and facilities. For example, there was a sudden increased pressure on the pharmacy department to replace all the nurses in administration IVs. This increased workload indicates the need to introduce additional structure-related KPIs such as patient-staff ratio to ensure that necessary improvements are made to structures as well.

According to the qualitative analysis, the improved outcomes is that they may not have been a result of the JCIA process, according to some participants. For example, the percentage of laboratory critical value reporting within 30 minutes already had high compliance of 99.37% prior to the JCIA process since they laboratory department already adhered to the College of American Pathologists (CAP) Laboratory Accreditation Program according to the lab specialists and quality department employees that were interviewed. Another example of an outcome that may not have improved due to receiving accreditation, according to a participant, is mortality rates. Mortality rates may not have improved as a result of pursuing the JCIA since its focus is more on improving the process, which might not translate to better patient outcomes, especially in a three-year period. This conclusion was supported by a quality department employee who stated that they did not have any performance improvement projects that targeted the reduction of patient mortality, which indicates that the JCIA may not have a direct impact on reducing them at KFHU.

The outcomes that did not show any improvements included: the rate of patients who left the ER without being seen, the percentage of OR cancelations, and the rate of patient falls. The qualitative analysis provided some context and reasons for the lack of improvements for these KPIs. One participant stated the increase in the number of patients leaving the ER increased because many patients use the emergency room to open a medical record and not to seek treatment, which led to the increase. After introducing the JCIA, the quality department increased awareness among nurses on the importance of reporting patient falls through an educational campaign, which caused an increase in reporting. Furthermore, a nurse and quality department employee stated that another potential reason for the increase in patient falls is the high nursing staff turnover rate, which led to an unstable workforce and an influx of new nurses with limited training.

Recommendations

One recommendation to the KFHU administration and the Joint Commission International is to include more locally relevant structure KPIs to better assess their impact on both process and outcomes and formulate more feasible solutions. For instance, an important KPI to introduce would be the nurse turnover rate since the qualitative analysis suggests that it might directly impact the rate of patient falls and possibly other patient safety outcomes. Moreover, to ensure more cooperation from health providers, the leadership should address the factors that cause resistance to participate in the JCIA process. Lastly, introducing additional relevant KPIs, such as patient-staff ratio and nursing staff turnover, can assist the hospital’s administration in formulating more informed interventions.

One of the main reasons for resistance is the lack of incentives for involvement in the accreditation process. Many participants felt that being pressured to participate and mandating compliance without any incentive created a lot of resistance. To address this issue, incentives such as recognition, awards, monetary compensation should be introduced to encourage administrative employees and health professionals at all levels to participate at higher levels. An additional recommendation is for the hospital’s leadership to make improvements in all departments’ work environments. Creating workspaces that encourage better teamwork can increase stakeholders’ participation and ultimately create more effective interventions.

Furthermore, several participants expressed their concern the some of the positive effects of the JCIA are unsustainable. Moreover, to mitigate the effects of a Hawthorne Effect in observation based KPIs is to use announced or unscheduled internal audits to ensure health professionals’ compliance. Lastly, this study identified issues regarding the JCIA program itself, including the JCIA surveyors and the JCIA standards. According to the participant’s responses, they thought the surveyors were not specific with their recommendations to improve certain outcomes, making their proposed solutions unfeasible at KFHU. For that reason, it is recommended that the JCIA takes into consideration encouraging their surveyors to familiarize themselves with each hospital’s unique organizational culture. The surveyors should also consider the hospital’s size, structures, and rescuers to ensure the specificity and the feasibility of their recommendations for improvements.

Study Limitations and Implications for Further Research

This study was limited to a single teaching hospital in Khobar, Saudi Arabia. Therefore, the results may not be generalized to other hospitals in different countries or with different settings, sizes, and services. Another limitation is that the dissertation research had 12 outcome measures, which were limited to process and outcome measures. Future research should include structural measures, such as the physician-to-patient ratio, to ensure the assessment of the impact of the JCIA process on all the domains of quality. Adding structure-related KPIs can also give some insight as to why specific outcomes did not improve. For instance, the lack of improvement in a particular outcome may not be due to an issue in compliance but due to limited personnel or resources. It is also important to note that despite excluding many KPIs from this study, it may not have skewed the assessment of the impact of the JCIA since the hospital did not make any interventions to outcomes they did not measure. However, some critical KPIs were excluded due to not being measured every month or were not collected throughout the period of study, such as sentinel events, which were collected three months after accreditation. The limited number of KPIs can be avoided in future studies since many hospitals in the Kingdom, including KFHU, are now more acquainted with the JCIA process and the importance of continuous monitoring and evaluation.

Additionally, a single time series may not have accounted for the possibility of seasonal effects, especially if the pre- and post-intervention months are uneven, which could create a bias in the results. This bias is caused by the unequal distribution of summer to winter months. Future research should adopt a model that is stratified by month using dummy variables for each month to account for seasonality. Another solution would be to use a control group referred to as a controlled, interrupted time-series design.

One of the qualitative analysis limitations was the limited involvement of some of the participants in the accreditation process. Most of the participants from the medical records and the radiology department noted that they were not involved from the beginning in the accreditation efforts; therefore, their responses did not provide a comprehensive view of its impact in relation to their work. Additionally, no interviews or surveys were used to understand the patients’ perspectives towards this process. Future studies should aim to investigate patients’ attitudes and perceptions towards JCIA accredited hospitals versus non-accredited hospitals since this information can assist decision-makers in improving the patient experience. Moreover, the qualitative analysis in this study indicated the need to investigate the associations between different outcomes, such as the potential link between the rates of hospital-acquired infections and hand hygiene compliance, to provide more insight on how the JCIA process itself led to better outcomes.

Lastly, the study period only included one accreditation cycle, limiting providers’ perceptions to a hospital going through the JCIA process for the first few times. For instance, provider resistance to participate in quality improvement efforts may decrease in future accreditation cycles due to the organizational culture shift to a culture that promotes patient safety. The extended study period may also help investigate whether the increased workload and resistance from health providers towards the accreditation process is a pervasive issue. Additionally, a more extended study period would be needed to investigate the impact of newly introduced KPIs at KFHU, such as the rate of medication errors and the number of near misses, which are the unplanned incidents that were close to causing harm or injury to patients. Moreover, there’s a need to investigate the organization-related lack of improvement in OR same-day cancellations to formulate effective solutions. For that reason, further research is necessary to investigate factors that influence the effectiveness of accreditation and its impact on hospitals that went through several accreditation cycles.

Conclusions

Many health organizations worldwide pursued the JCIA, despite the inconsistent evidence on its effectiveness. Among these countries was Saudi Arabia, which has the second-highest number of JCIA-accredited health organizations. For that reason, this study aimed to evaluate the effectiveness of the JCIA using KPIs and administrators and health providers’ perspectives. The results of this study indicated that the JCIA as a quality improvement tool was effective in improving most study outcomes. The researcher also identified factors that influence the JCIA process’s success, which included the health providers’ resistance to participate in the accreditation process. Such factors need to be further investigated and addressed by the hospital’s leadership to ensure a more efficient implementation of the JCIA in upcoming accreditation cycles.