Collecting Patient Feedback as a Means of Monitoring Patient Experience and Hospital Service Quality – Learning from a Government-led Initiative

Introduction

Since its inception in 1988, the concept of “patient-centered care” has identified eight characteristics as the most important indicators for improving quality and safety, two of which are respect for the patient’s values, preferences and expressed needs, and the involvement of family members and friends in the care process.¹ Patient feedback can promote patient participation, safeguard the patient's right to autonomy and shared decision-making, enhance communication and trust between doctors and patients, and contribute to achieving high quality care.^2,3

Patient participation has been recognized as a means of improving service quality as it contributes to identifying existing or potential medical problems and provides valuable first-hand experience for the development of medical services.^4,5 It can predict quality results, service effectiveness and efficiency.⁶ It can also effectively reduce patient injury and associated medical cost^7,8 and is emerging as an important measure of improved health outcomes.⁷ Actively seeking patient feedback on matters related to patient experience and expectations not only inform service improvement, but also improve patients’ trust, thus maximizing the appropriateness of services and the service improvement efforts.⁷ It is evident that the development of credited channels for patient feedback is important to allow patients to express their views and concerns, and seek help without the fear of being treated differently and unfavorably. Systematically analyzing patient complaints can prompt service providers to improve health services in a timely manner which can reduce medical errors and improve patient care process.^9,10 A good feedback system is a key enabler for the adoption of a patient-centered approach transforming patient and families’ participation in the diagnostic and treatment processes from being passive to active which not only improves service and patient outcomes, but also encourages higher level of compliance from patients in the treatment process. Ultimately, it contributes to the improvement of working relationships between patients, clinicians, and service providers.^11–13

The Patient Feedback System

Globally, various platforms have been adopted to enable patient feedback on a broad scale, such as the centralized complaint handling service by the National Health Service in the UK;¹⁴ the consumer assessment of healthcare providers and systems¹⁵ for patient complaint data platform (PCDP) and Center for Patient and Professional Advocacy (CPPA) for patient advocacy reporting system in the US,¹⁶ and the formal patient complaint channels established by the health complaints commissions (HCCs) and the Australian Health Practitioners Regulation Agency in Australia.¹⁷ However, the inaccessibility or inappropriateness of patient feedback tools and the tension caused by unreasonable handling of patient feedback can suppress the participation and authenticity of patient feedback. This may present a “feedback fatigue” that results in ineffective communication and engagement with patient feedback.¹⁸ Feedback reported to the central system independent of health service providers can increase patients’ confidence in the protection and respect of their confidentiality and ensure the reliability of the data collected, hence maximizing the value of patient participation.^6,19

Using Patient Feedback to Guide Service Improvement

Turning complaints data into useful information that can guide service improvement could be another challenge when the text is unstructured, multifaceted, and existing as a free narrative. The process of investigating and translating complaints data into meaningful information are complex, time-consuming, and labor intensive.²⁰ To prevent using nonuniform standard research methods that affect the reliability and effectiveness of research results, Gillespie and Reader²¹ developed the health care complaint analysis tool (HCAT) where its reliability and accuracy has been tested. HCAT has since been successfully applied in different regions and countries in categorizing and analyzing patient complaint/feedback data.^22–26 HCAT categories patient complaints into three main domains: clinical, management, and relationship. Each domain is further categorized into two or three subcategories: quality and safety for the clinical domain, environment and institution for the management domain and listening, communication, and respect and patient rights for the relationship domain.²¹

Patient Feedback Improvement in China

Building trust and improving doctor–patient relationships have strategic importance in China.² In the notice on further strengthening patient safety management, the Chinese government requires the construction of a “government-led, social collaboration, public participation” pattern of patient safety work, to encourage patients to participate in patient safety activities. A Chinese government owned public service platform for various non-emergency appeals, namely the “12345” Government Service Convenience Hotline (GSCH) was first established in 1983 in Shenyang, the capital city of China’s northeast Liaoning Province, then slowly introduced across China becoming a consistent feedback channel in all major cities in recent years.²¹ The GSCH is a platform that accepts all kinds of nonurgent appeals including patient feedback. Under the guidance of the General Office of the State Council, it was unified as the “12345” hotline, providing “7×24” all-weather manual service.²⁷

In the medical field, GSCH encourages patients and their families to express views and concerns, share experience in the patient journey, and make suggestions which will then be provided to relevant health services or departments for consideration and action in a confidential and timely manner under strict guideline.²⁷ The GSCH will then follow up with callers to report on the associating actions addressing their complaints and to obtain feedback on their experience and satisfaction of the associating actions taken by the health services/hospitals/departments as a response to their initial complaints made via the hotline. This process ensures that patient's feedback is valued which further encourages the patient's willingness in participating in the care processes in order to improve the quality provision of medical services.

Although the feedback hotline has been open for residents in Shandong Province for more than a decade, the GSCH has only been fully functioning covering all major cities from 2017 and all regional cities from 2018 after significant efforts had been put into unifying and standardizing the hotline and promoting it for broader access. The improvement coincided with the quality improvement initiatives implemented in one of the major tertiary general university-affiliated hospitals (the hospital) in Jinan – capital city in Shandong province. To be compliant with the requirements of “Internet + Medical Health” of the Health Commission, the hospital began to gradually build its own smart service brand since 2018 with more than 10 initiatives successfully implemented across the hospital such as implementing an online appointment, application and initial diagnosis platform, implementing a multi-channel payment system, and digitized patient results from diagnostic testing and examination. An intelligent pandemic prevention access system has also been developed to ensure medical safety of patients since the COVID-19 outbreak.

By analyzing the patient complaint data concerning the hospital received via GSCH between 2016 and 2020, this paper aims to answer the following questions:

1. Have the efforts in unifying the GSCH and broadening its coverage significantly encouraged feedback provided by patients and families?
2. How have quality improvement initiatives implemented in the tertiary hospital affected the number and types of complaints made by patients and their families via GSCH?

By answering the above questions, factors that can encourage feedback from patients and families for service improvement purposes maybe identified to guide health care organizations in improving the overall experience of patients and the quality of services that they provide.

Materials and Methods

This study retrospectively analyzed all 4,021 complaints against the hospital in Shandong Province, China through GSCH from 2016 to 2020, including 2,688 patient complaints, of which 1,630 were related to patient care process and 1,058 were not.

Setting

The hospital, where patient complaint data was collected, services an area of more than 70,000 square meters with 95 clinical and medical technology departments, 3,217 inpatient beds with nearly 4,000 employees. Patient feedback hotline data from GSCH underwent a series of procedures such as standardized acceptance, immediate referral, time-limited handling and satisfaction evaluation. During the whole process, personal information and feedback of all callers are recorded and stored in electronic text files by the operator, and the callers were asked whether their personal information should be kept confidential.

Coding and Analysis

All complaints recorded from GSCH were carefully reviewed. Data related to patient care process in the hospital were coded based on the HCAT domains (clinical, management, relationship), levels of severity (low, medium, high), level of harms (none, minimal, minor, moderate, major, catastrophic), and hospital departments that the complaints focused on. Harm and severity are independent of each other. For example, a patient who describes a serious safety problem (such as a diagnostic error) may not be seriously hurt by the error. Complaints against staff member were further extracted and analyzed based on the following types: doctors, nursing staff, medical technicians, hospital logistics staff, auxiliary medical personnel, and hospital administrators/managers. Some information of the callers such as gender, whether they are patients or relatives of the patients who received the hospital services was also captured and analyzed. Data that are not related to care processes are separately coded based on the following services that the complaints concerned: hospital security services, parking services, fee and charges, and others.

Reliability of Data Coding and Analysis

For effective and reliable analysis of the data related to patient care process, the researcher and author SH received online training and participated in the test provided via the HCAT website, and then independently coded all patient complaints data. All researchers discussed the difficulties and differences encountered in the coding process, before finally reaching a consensus. Ten percent of the patient complaints data were coded again after a period of time. Inter-rater reliability was accepted in category (κ=0.795, P<0.001), in severity (κ=0.814, P<0.001) and in harm (κ=0.919, P<0.001) by Cohen’s kappa statistic. Besides, the complaint data of non-patient care processes were extracted into four main categories according to the complaint content after discussion. Coded data was cross-checked by senior author ZL for accuracy.

Data Analysis

First, we described a change in the amount of complaints data obtained from the GSCH and made a prediction analysis using the time series model named ARIMA. It combined both autoregressive (AR) modelling and moving average (MA) modelling. The optimal model was selected based on the values of the Akaike information criterion (AIC) and the residual analysis was completed. The auto ARIMA function package in R software could quickly find the appropriate model, and Ljung–Box test could confirm that the temporal autocorrelation no longer existed in the model residuals (P>0.05). Second, for data related to the patient care process, after being coded based on HCAT classification, was sorted through Microsoft Excel spreadsheets. Descriptive analysis was conducted to investigate the details about the domain, category, sub-category, severity, harm, stage, and background characteristics. Third, data not related to the patient care process were coded based on types of services of concern and sorted through Microsoft Excel spreadsheets. A descriptive analysis method was also performed. Partitions of chi-squared test is used to test whether the proportion change of all HCAT or non-HCAT data is significant (α’=α/(k+1), k=comparison times).

Fourth, mosaic plots were used to present the interaction relationship between different classification variables and using Pearson’s chi-squared test of independence for the HCAT data. Every rectangular area is determined by the frequency of the data recoded, and the greater the frequency, the larger the area. Solid rectangular area boundaries correspond to more observations than expected and fewer dotted lines. The depth of the color is determined by Pearson’s chi-squared test residual value. The absolute Pearson’s chi-squared test residual value of gray is less than 2, light color is 2–4, and dark color is greater than 4. Two-sided probabilities that the absolute values of the residuals are 2 and 4 approximately corresponding to 0.05 and 0.0001, respectively. This used the code written by Gillespie and Reader.⁵

The statistical analyses were performed using R software, version 4.2.0 (R Foundation for Statistical Computing).

Ethics Approval

This study was approved by the Medical Ethics Committee of the Second Affiliated Hospital of Shandong First Medical University (file number: 2022–093). This study strictly adhered to the ethical standards of the Declaration of Helsinki. All data collected are also subject to the data protection and privacy policy implemented by the university approved the ethic application of the project. No identifiable information/data was stored and data are only accessible by the relevant research team members of the project.

Results

Between 1 January 2016 and 31 December 2020, a total of 4,021 calls concerning the hospital were made via the hotline by patients and families. Close to 67% of these calls were classified as a complaint (refer to Table 1).

Table 1 Nature of Calls to Patient Hotline

During the study period, an average of 21.26 “complaint calls” were received per 100,000 patients of the hospital, ranging from 3.69/100,000 in 2016 to 47.31/100,000 in 2020. The annual complaints received between 2016 and 2020 are detailed in Table 2.

Table 2 Types of Complaints Received Each Year Between 2016 and 2020

Quantity and Variation

To understand the trends in patient complaints, monthly change in the number of complaints was plotted to predict out-of-sample patient complaints over the next 12 months using an ARIMA model for HCAT data (Figure 1) and non-HCAT data (Figure 2).

Figure 1 Patient care process complaints forecast plot for GSCH over the next 12 months using the ARIMA model with 95% and 99% confidence interval for dark and light blue colors, respectively.

Figure 2 Non-patient care process complaints forecast plot for GSCH over the next 12 months using the ARIMA model with 95% and 99% confidence interval for dark and light blue colors, respectively.

Both model specifications determined were ARIMA (2, 1, 0)×(1, 0, 0)₁₂. The residuals of ARIMA in Figure 1 (q = 15.183, df = 9, P = 0.086) and Figure 2 (q = 9.468, df = 9, P = 0.395) were stationary by Ljung–Box test. Figure 1 showed that since the start of development of the GSCH platform, the data showed a clear upward trend in the number of patient complaints, only interrupted by the sudden drop when COVID-19 broke out suddenly. Consistent with changing trends in hospital outpatient and inpatient numbers, complaints tend to be low during the spring Festival and rebound rapidly after holiday peaks, as predicted by the model. According to the simulation results in Figure 2, the number of complaints related to nonmedical procedures were persistently high in 2021.

General Information

Among the callers who made the complaints, approximately 29% of them were patients (n=788), the rest were patients’ children or other family members either making complaints on the patient's behalf or on their own experience and/or problems encountered. The gender ratio of patient callers was 1:0.74 for male and female respectively, which is different from the patient gender ratio of 1:1.04 of the hospital. The two departments received the most complaints were the clinical department (36.94%, n=1006) and hospital logistics department (32.06%, n=873).

Table 3 summarizes the total number of complaints made under each category in two time periods 2017–2018 vs 2019–2020. The proportion of the complaints for each category of the HCAT or non-HCAT complaints were also included. The changes in these two time periods for each category were also analyzed by partitions of chi-squared method and included in the table.

Table 3 Types of Complaints Based on HCAT Classification

There has been consist increase in calls in all three HCAT domains (χ²=15.282, P<0.001) and non-HCAT data (χ²=40.066, P<0.001). Amongst all the complaints related to patient care process, the proportion of complaints concerning management problems increased from 47.73% in 2017–2018 to 58.50% in 2019–2020 (α’=0.0125, P<0.001), whilst the proportion of relationship problems decreased from 33.65% to 25.69% (α’=0.0125, P=0.002). The proportion of the complaint concerning security services increased from 1.94% in 2017–2018 to 12.73% in 2019–2020 (α’=0.01, P<0.001), and fee and charges from 26.70% in 2017–2018 to 15.67% in 2019–2020 (α’=0.01, P<0.001) were statistically significant. The proportion of car-park-related complaints also continued to increase (50.00% vs 58.87%). These changes were consistent with the adjustment of the parking fee charges and parking time limit at the hospital car park, and the introduction of visitation restriction since the COVD-19 outbreak.

Based on the HCAT complaint classification, Figure 3 provides a visual trend in the changing number of complaints received for each category between 2016 and 2020. Figure 4 shows the proportion of different complaint subcategories in the two periods of 2017 and 2018 vs 2019 and 2020.

Figure 3 The changing number of complaints received for each category between 2016 and 2020.

Figure 4 Proportion of different complaint subcategories in the two two-year periods.

Institution issues received the most complaints with consistent annual increase, mainly about the unexpected cancellation and rescheduling of appointments (eg, “Do physical examination in the health examination center of the hospital, and be informed that the examination doctor leaves work and needs to come for physical examination the next day…”), and being declined an appointment (eg, “My brother is treated in the emergency intensive care unit of the hospital. He is in stable condition and can be transferred to the immunology department. However, the Director of the immunology department said that the beds in the immunology department were full and did not accept patients, and asked us to contact other hospitals for transfer…”). Complaints of “environment” issues continued to increase, exceeding “respect and patient rights” in 2019 and 2020, after “institution” issues. (Figure 3). A significant increase in environment-ralated and institution-related complaints were recorded in the 2019–2020 two-year period after the hospital introduced a series of quality improvement initiatives in late 2018 as mentioned in the introduction. This contrasts with the slight increase in the complaints concerning quality problems and the decrease concerning problems in the remaining other subcategories (Figure 4).

Table 4 details types and gender of the callers who made the patient care process-related complaints (HCAT data) and the three key departments and three key types of hospital staff that the complaints were concerned about. Close to 50% of the patient care process related complaints were made against specific staff, in particular doctors (55.87% of all complaints against staff), nursing staff (19.57%), and medical technicians (10.32%).

Table 4 Characteristics and Focus of the Complaints

Figure 5 shows proportion of annual patient care process related complaints against three types of hospital personnel between 2016 and 2020 with doctors receiving more than half of the complaints.

Figure 5 Proportion of complaints against staff.

Using the HCAT data, Figure 6 and Figure 7 show the proportion of complaints in three different severity levels: low, medium and high and six categories of harm: none-minor (combining three categories of none, minimal and minor), moderate, major and catastrophic in mosaic plots and results of Pearson’s chi-squared test of independence.

Figure 6 Three types patient care process related complaints and severity.a Notes: aRectangular area indicates the frequency of complaints in each domain with different severity. The number indicates the complaints percentage of rectangular area in the same column and the frequency in brackets. Solid rectangular area boundaries correspond to more observations than expected and fewer dotted lines. The absolute Pearson’s chi-squared test residual value of gray is <2 (P>0.05), light color is 2–4 (P<0.05), and dark color is >4 (P<0.0001).

Figure 7 Three types patient care process related complaints and harm.a Notes: aRectangular area indicates the frequency of complaints in each domain with different harm. The number indicates the complaints percentage of rectangular area in the same column and the frequency in brackets. Solid rectangular area boundaries correspond to more observations than expected and fewer dotted lines. The absolute Pearson’s chi-squared test residual value of gray is <2 (P>0.05), light color is 2–4 (P<0.05), and dark color is >4 (P < 0.0001).

Table 5 provides a comparison on the number and proportion of the level of harm associating with the HCAT data between two two-year periods: 2017–2018 and 2019–2020 under three HCAT categories.

Table 5 Level of Harm vs Patient Care Processes Related Complaints in Three Categories in two Periods

Discussion

Efforts in Encouraging the Access of Patient Feedback Hotline

Patient feedback, negative or positive, provides important forms of evidence on service quality which can be used by health-care organizations in preventing and reducing medical errors, adverse events, hence reducing patient disputes and medical malpractice claims.^28,29 Therefore, efforts in creating effective feedback system and encouraging the provision of feedback by patients and their families can ultimately contribute to service quality improvement.³⁰ The five-year data collected from a feedback hotline on a major tertiary general university-affiliated hospital in China confirmed a significant annual increase in the number of complaints received that are directly or indirectly related to the patient care process, demonstrating the increasing confidence of the general public in the integrity of the government owned and operated service and related processes. This is a particularly important step toward addressing the widespread concerns of hospital service quality, increasing medical disputes, and tense doctor–patient relationships in China.³¹ The increasing trend of accessing the hotline also proves that the government’s efforts in encouraging patient feedback have been effective and the important roles that government should play in hospital service quality and safety improvement. Adopting a unified process with clear operational guideline, operated by well-trained hotline staff with a professional and nonjudgmental manner, providing fast referral to hospitals for actions and timely update to callers were the key to such success. Patient feedback can be obtained via many means, but fundamentally, a good feedback system should provide a clear and easy to follow process allowing actions to be taken in a timely manner.³²

Learnings from the experience of the GSCH platform demonstrated that the success of a public feedback system should have the following key elements as illustrated in Figure 8.

Figure 8 Framework of a successful public feedback system.

It starts with a clear purpose – collecting public views to guide service improvement. Information collected should be used to guide improvement in the system and organization level rather than laying the blame on individuals. A no-blame culture not only enables a meaningful and effective feedback process, but also improves staff morale and clinician–patient relationships.³³ A clear feedback loop and having well-trained staff in managing the feedback system are also critical. The success in the GSCH hotline also confirms the importance of two-way feedback system – those who made the calls receiving feedback on actions taken in addressing their concerns in a timely manner. Making patients and their families feel that their feedback is valued and can lead to service improvement is a key motivator for meaningful and genuine feedback to be provided.³³ Important factors also include ease of and convenient access, interactive support and scientific implementation principles.³⁴

The Value of Government Led Feedback System

Patient feedback provided during the diagnostic and treatment process is often spontaneous and influenced by one’s own emotions which may be perceived as biased against clinicians and health-care organizations.³⁵ In contrast, patients may be reluctant to provide feedback out of fear of unfavorable “treatment” and undesirable consequences.³⁶ The existing internal feedback processes in hospitals, commonly known as patient satisfaction survey during the patient’s stay and upon discharge are not effective in addressing these challenges. Therefore, a confidential and unified feedback system independent of the hospitals/health services perceived as nonjudgmental, transparent and fair such as GSCH will maximize the contribution from patients and families in sharing their experience, thoughts and suggestions. The success of GSCH further confirmed the efficiency of government’s efforts in vigorously integrating broad range of nonemergency hotlines into one single hotline and acting as a third-party “operator” in navigating a caller’s feedback loop. This has successfully ensured that feedbacks are addressed and acted upon by the relevant parties in a timely manner. Again, the rapidly growing patient feedback data confirms that the GSCH platform has effectively encouraged patients to express their needs, increased patients’ trust, and successfully engaged patients' and families’ participation in service improvement.

As mentioned in the introduction, many complaints against hospitals and clinicians in China were the results of mistrust which could have been prevented when effective dialogues are encouraged and communication channels are provided.³⁷ By providing a nonjudgmental, transparent and confidential system such as GSCH to patients and families allowing them to express their concerns to well-trained operating staff, may ultimately develop trust and reduce frustration, hence reducing the number of preventable disputes and malpractice claims and violence against clinicians. However, it effectively demonstrates to the public the system level efforts in improving hospitals/health services processes and the commitment from both government and health-care organizations in creating more positive patient experience and providing services that are safe and of high quality. Developing patients’ trust in health services relies on all level of efforts.

Creating Better Patient Experience and Improving Service Quality

The five-year GSCH data confirms the ability in using patient feedback to confirm service quality and effectiveness of service improvement strategies. This study shows a decrease in the proportion of complaints regarding clinical and relationship issues in the two-year period of 2019–2020 in comparison with 2017–2018. The decrease coincides with the various initiatives implemented by the hospital in streamlining service process and service quality improvement since 2018. It confirms the reliability of patient complaint data collected via GSCH and that efforts in improving patient care processes can lead to a better patient experience and higher service quality.³⁸ Conversely, the increase of proportion of complaints concerning management problems indicates that the hospital service improvement strategies have not been effective in addressing management deficiency such as facilities, clinical equipment, staffing level, waiting times, and access of care. However, such increase may be partially attributed to the COVID-19 pandemic when sudden increase of service demands, change of service delivery model and routine, deferring and restricting specific treatment and services, and modifying clinical pathways were witnessed.³⁹ Similar to lessons learnt globally, hospitals and health services need to continue their efforts in developing their surge capacity and crisis management strategies.⁴⁰

Improving Patient Safety and Systemic Improvements

A comparison of the level of harm associating with patient care process related complaints between 2017–2018 and 2019–2020 revealed that there is no decrease in the proportion of patient care process-related complaints that are in the major and catastrophic level. Although proportion of patient harm at a catastrophic level during the two two-year periods are not significantly higher than the results from international studies on patient complaints using the same HCAT tool as detailed in Table 6, it is alarming that more than 14% of all clinical problems are causing patient harm at a catastrophic level.

Table 6 International Studies on Patient Care Process Related Complaints with Harm in Catastrophic Level

Despite efforts in improving the quality and safety of patient care, patients are still receiving services from public hospitals that may cause them harm. The dissatisfaction and negative experience of hospital services are still vastly related to clinicians involving doctors and nurses. This reinforces that significant efforts are required both from government and hospitals in improving the relationship between clinicians and patients.⁴² Such efforts may include further developing the professional identify of clinicians, improving communications between clinicians and patients,^2,43 and developing transparency and a no-blame culture further encouraging feedbacks from both hospital staff, patients and their families.⁴⁴ To enable this, the review of the process of responding to and resolving complaints and addressing identified medical errors and oversights is necessary to ensure that hospitals are taking responsibilities in not only protecting their patients, but also their staff. In addition, effective clinical governance should be developed to encourage all levels of staff including senior management sharing the responsibilities in ensuring high quality and safe services are provided.⁴⁵

Inviting patient feedback on service quality is important, so is involving patient and families in the design of service improvement strategies. The significant increase in the complaints in relation to institutional services and processes indicate that the hospital may have failed to consider patients’ preferences and difficulties encountered during the process of receiving care. Therefore, in order to improve the relevance of service improvement initiatives to overall patient experience and service quality, before implementing service improvement initiatives, hospitals should develop a process allowing patient’s views to be sought and considered. However, this may challenge the traditional centralized decision-making and top-down leadership approach adopted in the Chinese public system.⁴⁶ Therefore, the patient feedback hotline has become a major channel to informing improvement of patient care process with a patient-centered approach. The continuation of using the hotline can be seen as a means to monitor hospital service quality and predict the benefits of quality improvement initiatives. However, process should also be introduced to specifically seek feedback on outcome and impact of service improvement initiatives from both patient and service providers’ perspective. Research on the motivation and satisfaction of the people who provide feedback via the hotline will inform the improvement of the effectiveness and efficiency of online feedback services.

Conclusion

The analysis of the five-year complaint data collected via the GSCH platform in China provides valuable insights into one of the effective means for health-care organizations in seeking patient feedback to guide the improvement of the overall patient experience and the quality of services. It confirms the advantages of having a system that is nonjudgmental and transparent and that protects confidentiality in encouraging patient feedback. It reinforces how health services can utilize patient feedback to monitor the performance of their staff and services which may guide service improvement and the creation of a positive patient journey. The findings from the success in GSCH provide useful guidance in developing an effective and comprehensive feedback system in other health-care context. The study has proven the value of patient participation and value for health service providers and their staff in actively seeking and considering patient feedback for the purposes of improving patient care processes.