Towards Better Perioperative Pain Management in Mexico: A Study in a Network of Hospitals Using Quality Improvement Methods from PAIN OUT

Introduction

Unrelieved post-surgical pain is a major, unsolved healthcare problem, worldwide. Studies from high resource countries such as the United States and Europe indicate that patients report a high incidence of moderate to severe pain after surgery.¹ The pain is associated with short- and long-term negative effects on patients’ function and quality of life.^2,3 The considerable variability in pain management practices and outcomes observed across institutions indicates that care is not optimal and that guidelines are often not implemented in the clinical routine.⁴ Less is known about care and outcomes in middle-income countries. Garcia et al⁵ reviewed perioperative pain management and outcomes in Latin America, including studies from Mexico, Chile, Brazil Colombia and Peru. These report findings which are similar to those observed in high resource countries.

Clinicians, basic and clinical researchers and policy makers have made major attempts over the last 50 years to improve perioperative pain management at the local, national and international levels. Attempts include advocacy and policy making, developing tools for education, carrying out basic and clinical research leading to the development of clinical practice guidelines, and establishing structures within hospitals of specialized teams to provide care, such as Acute Pain Services.^6–10

The research addressing how to improve healthcare is known under a variety of names such as “quality improvement (QI)”, “implementation research” or “knowledge transfer”.¹¹ Activities to improve healthcare quality and safety include audits, findings that are fed-back to clinicians, researchers and decision-makers, and organizational change, involving skill teaching, developing and implementing local treatment protocols.¹² Work on the basis of single centers has often not been effective or enduring and was, therefore, an incentive for multi-disciplinary teams from different organizations to join forces and work together over a period of months to several years in a structured way as ‘QI collaboratives’. This trend has been evident in the US, Canada, Australia and Europe since the late 1980s.¹³ Collaboratives have included several and up to 300 hospitals.¹⁴

In 2009, a team of clinicians and researchers sought a new approach to improve quality of care in the perioperative pain management setting by establishing PAIN OUT, an international, quality improvement and research network with a multi-center registry (www.pain-out.eu). The program offers a platform for standardized web-based auditing and feedback for assessing pain management and Patient Reported Outcomes (PROs) in the clinical routine. Initially, hospitals taking part in PAIN OUT carried out QI by working on a single-center basis. However, over time it became evident that the effectiveness of QI may be increased by creating a collaborative, or a “network”. Consequently, the objective of this project was to carry out a pre–post study in up to 10 hospitals in Mexico, over a period oftwo years, during which staff in each ward, under the leadership of local colleagues (“network leaders”), would develop and implement QI interventions related to perioperative pain management. Staff would use tools developed by PAIN OUT and assess the following features: (i) technical, whether the network would be able to develop and implement strategies for QI and collect patient data; and (ii) clinical, whether treatment processes and PROs would change within the network, with a focus on worst pain ratings as a primary outcome. We describe findings from this project here.

Methods

The team coordinating PAIN OUT, at the University Hospital Jena, took responsibility for overseeing the project. The PAIN OUT methodology is registered with the US National Library of Medicine (ClinicalTrials.gov NCT02083835). Pfizer provided an un-restricted educational grant to cover the cost of the annual subscription to PAIN OUT for up to 10 hospitals over a two-year period, and remuneration to each hospital for datasets collected.

Setting Up the Network

PAIN OUT offered the role of network leader to A.G and V.A. in light of their experience with QI activities related to pain care and that the institution in which they work plays a leading role in clinical research in Mexico and internationally. A.G and V.A. approached potential principal investigators (PIs) in different hospitals asking that they take part in the project. PIs could be anesthesiologists or surgeons, willing to participate in a two-year project and have the human resources to carry out the project. Additionally, they would create a multidisciplinary QI improvement working group, charged with developing and implementing the intervention. Lastly, participating centers would be geographically close so that physical meetings could take place on a regular basis without requiring considerable travel time or expense.

Study Design and Stages

This was a quasi-experimental, pre–post-study design,¹⁵ for implementation in one or two wards in the network hospitals. PIs were free to select the surgical discipline(s) of the wards where they would carry out the project and the nature of the QI intervention. An administrative phase of 6 months preceded the study, during which PIs in each hospital arranged the contract with PAIN OUT, obtained approval for collecting non-identified patient data from the local ethics committee in accordance with the Helsinki Declaration (the details are listed in the Supplementary data files, Table S-1) and assured that 2–3 surveyors studied the project’s methodology for collecting data,¹⁶ as described below.

The study stages consisted of:

Phase 1: a) An introductory workshop for participants to review the project’s aims and methodology; b) Collection of baseline data, 80–120 datasets per/ward; and c) Hold a mid-project workshop to discuss baseline findings, using the findings as a basis for proposing the QI measures.

Phase 2: Working groups in each hospital selected the QI measure(s) and worked towards implementing them.

Phase 3: A second round of data collection took place in the same 1–2 wards (80–120 datasets per/ward), followed by data analysis and preparation for the summary workshop.

Phase 4: A summary workshop took place during which participants presented findings from the project and planned whether and how to carry out follow-up work within each hospital and/or expand the work to additional hospitals.

PAIN OUT allowed leeway for the duration of the phases; however, we assured that all hospitals in the network would progress together from one phase to another and aimed that the project would be complete within 2 years.

Patient Recruitment and Measures for Studying Processes and Outcomes of the Interventions

Patients could be enrolled if they fulfilled the following inclusion criteria: (1) were of consenting age, 18 years or older; (2) were on the first postoperative day (POD1) and back on the ward from the post-anesthesia care unit for at least six hours; and (3) agreed to participate in the survey. Consent could be oral or written, depending on the requirements of the local ethics committees.

Variables collected for each patient involved:

(1) Demographic and perioperative process data included items such as patient gender, age (year of birth), weight, height, analgesics administered (as pre-medication, intra-operative, post-anesthesia care unit and ward), type of surgery (using the International Classification of Disease procedure codes, [ICD9]), method of anesthesia, and whether there was a record of evaluating pain intensity in the patient’s chart at least once since surgery. A study surveyor obtained this information from the patient’s medical record.

(2) International Pain Outcomes Questionnaire (IPO-Q)¹⁷ consists of 13 items evaluating four domains: (a) intensity of pain; (b) interference of pain with activities in and out of bed and with negative affect; (c) adverse effects associated with the anesthesia and treatment with opioids; and (d) perception of care. Patients assess most items using an 11-point numerical rating scale (NRS, 0 = no sensation, 10 = worst possible pain). Some questions require dichotomous yes/no replies and others, a percentage scale (0–100%). Patients also indicated whether they received or used non-pharmacological method to relieve their pain after surgery. Patients related all these questions to the time since their surgery. Lastly, patients reported whether they experienced a persistent painful condition for at least 3 months before surgery and its intensity. The IPO-Q is validated in English and has since been translated into 28 languages. Patients were offered a version in Mexican Spanish.

Structural data addressed the type of hospital, teaching status and number of beds in the hospital. The PIs provided the information. Data collection, management and storage Surveyors in each hospital underwent training for collecting data and approaching patients. This included reading a manual, filling in a quiz and submitting test data sets, which were audited for completeness and accuracy of data entry. Surveyors entered the data into a web-based, password secure portal. The PAIN OUT database is hosted at the Jena University Hospital.

Plan for the Evaluation

We assessed the feasibility of using QI methods to change practices and PROs within the network of hospitals by evaluating the following features:

Technical

(1) The number of hospitals completing the project and patients recruited during the two project phases.

(2) The proportion of missing data collected at baseline and post-intervention for key variables. The rate of missing variables is a marker for the feasibility of data collection and less than 5% data loss is regarded as inconsequential for data analysis and concerns about bias.¹⁸

(3) Qualitative description of the QI interventions and evaluation of the program’s effect on the staff and patients. PIs filled in a semi-structured questionnaire listing the intervention(s) they carried out, evaluating the usefulness of the project to the staff, and to patients, as well as barriers and challenges related to carrying out the project. Lastly, they stated whether they would wish to continue and upscale the project. Here, we briefly describe these findings.

Clinical Features of Implementing the Project

The changes were evaluated on two levels, that of the whole network and on a ward by ward basis. PAIN OUT and the teams in Mexico deliberated on contemporary guidelines and consented on the following variables, or “quality indicators”, for the analysis.

(1) For evaluating change in PROs, we selected 1–2 variables from each of the four domains in the IPO-Q and a threshold above which clinical intervention may be recommended:^19–22 (i) pain intensity: worst pain since surgery ≥6/10; percent time in severe pain ≥30%; (ii) interference of pain with activities in bed (eg sitting up or turning) ≥4/10; (iii) adverse effect: nausea ≥4/10; (iv) perception of care: answered wish more treatment for pain with “yes”. Lastly, we formed a composite measure, whereby, a patient qualified as having “high pain and high interference” if he/she reported worst pain ≥6/10 and were ≥30% of the time in severe pain and reported ≥4/10 in at least one of the followings: pain interference with sleep or with breathing and coughing or with movement in bed and/or out of bed ≥4/10. The proportion of patients, in each ward, whose evaluations were above these thresholds, was determined for the baseline and post-intervention phases.

(2) For evaluating change in treatment processes, we included interventions recommended for most patients undergoing surgery, as part of a multi-modal analgesic regimen ²³ and that are largely independent of surgery type. This included evaluating the proportion of cases where providers: (i) offered patients’ information about pain treatment options; (ii) infiltrated the surgical wound intra-operatively; (iii) administered a non-opioid; and (iv) assessed pain once patients were back on the ward.

The network leaders wished that the QI program should address the utilization of regional anesthesia, and so the use of this technique was also included in the evaluation.

Non-pharmacological treatments are considered a cornerstone for the management of pain after surgery ²³ and were, therefore, also included in the evaluation.

(3) Seeking associations between PROs and processes. The analyses in Clinical Features of Implementing the Project (1) and (2) would indicate whether the project brought about change in these variables and the extent of change. We also included an evaluation as to which of the PROs might be associated with the treatment processes and the magnitude of this association.

Primary and secondary outcomes. The primary outcome was the difference in the proportion of patients reporting ‘worst pain ≥ 6/10ʹ between the two project phases. The remaining PRO and process indicators served as secondary outcomes.

Statistical Analysis

The analysis was performed with data from wards contributing ≥30 data sets, per project phase.

Descriptive Analysis

Patient-related data (age, gender, existence of comorbidities, information on pre-existing pain, opioid administration before admission) and surgery-related data (ICD-9 procedure code, duration of surgery) are presented for the whole network. We report absolute and relative frequencies for dichotomous data and median, first (Q1) and third (Q3) quartiles for continuous data. The distribution of missing values for all PROs and process indicators was analyzed on the network and ward level.

Quality Improvement at the Network Level

To account for the clustered structure of the data and the dichotomous nature of the primary outcome (worst pain rating ≥6/10 NRS), we used the modified Poisson regression approach to obtain Relative Risk estimates (RR).^24–26 In detail, dichotomized worst pain ratings served as the dependent variable and the project phase was entered as the independent variable in the model. The resulting regression coefficient for the project phase (transformed to RR), including the 95% confidence interval, was used to evaluate the effect of the quality improvement. Additionally, age, gender and pre-existing pain were entered as covariates into the model. In a secondary analysis, the same method was used to evaluate the effect of the quality improvement interventions for the remaining PROs. As a measure of clinical effect size and based on the obtained models, we report adjusted absolute risk differences between project phases 1 and 2. Finally, similar models, without covariate adjustment, were obtained for the process variables. RRs can be interpreted in the following manner: a RR of 1 indicates no change, a RR < 1 indicates a decrease and a RR > 1 indicates a percentage increase in the corresponding PRO or process variable between project phases.

Quality Improvement at the Single Ward Level

Relative frequencies of the PROs and processes were compared between both project phases using Fisher’s exact tests. The effect size was evaluated based on φ coefficients with the following interpretation of absolute values: 0.10 to 0.29 small, 0.30 to 0.49 medium and ≥0.50 large.²⁷

Associations Between PROs and Processes

The second series of regression analyses, using a similar approach as described above, was carried out to evaluate associations between PROs (dependent variables) and process variables (independent variables). Effects of the process variables were separately modeled on the patient level (eg whether an individual patient reported receiving treatment information) and on the ward level (eg the proportion of treatment information on the ward at the corresponding project phase). The models were controlled for age, gender and pre-existing pain. Due to multiple testing, p-values for regression coefficients of process variables were adjusted using the Bonferroni-Holm method.

For the analysis we used R, Version 3.5.1, R Foundation for Statistical Computing, Vienna, Austria and SPSS (Version 22, IBM, USA). In all analyses, we applied a significance level of 5% and report two-sided p values. Recommendations by Bogduk et al ²⁸ guided the writing of this report.

Results

Technical Features of Implementing the Project

(1) Hospital and patient recruitment. The project was carried out between July 2016 and December 2018. PIs from 16 hospitals in Mexico City were approached to join. Of these, PIs and surveyors in 10 hospitals completed the administrative phase and nine completed the project. Eight of the hospitals were publically funded and one was private. The mean number of beds in each hospital was 384, ranging from 250 to 900. PIs included nine anesthesiologists and one orthopedic surgeon. The extra six months beyond the planned two years were a consequence of the 2017 earthquake in Mexico which caused considerable damage to several participating hospitals. As a result, work related to the project ceased for some months and collaborators from one hospital had to discontinue their participation in the project.

Figure 1 depicts the process of recruiting patients from the participating hospitals and wards during the two data collection phases. In the final analysis sample, wards from the following surgical discipline took part: general surgery (n=6), orthopedic and spine (n=3), thoracic (n= 2), urology (n=1), gynecology (n=1). The surgical procedures most commonly carried out for general surgery were: laparoscopic gastroenterostomy and laparoscopic cholecystectomy; for orthopedic surgery: total knee and hip replacement; for thoracic surgery: thoracoscopic excision of lesion and tissue of lung and lobectomy of lung; for urology: transurethral removal of obstruction from ureter and renal pelvis; for obstetrics and gynecology: radical abdominal hysterectomy.

Figure 1 The flow chart depicts patient recruitment during the two project phases.

Patient demographics, including information about medical history related to management of acute pain, are listed in Table 1.

Table 1 Patient Demographics. Findings are Shown Separately for the Two Study Phases

(2) Missings. As a general rule, the number of missing values was low. The total median percent was below 1% for the majority of PRO, process and demographic variables used in the analysis (for details see Supplementary data files, Table S-2).

(3) Qualitative description of the QI interventions. In the first phase of the project, collaborators considered that they lacked sufficient knowledge to provide care and teaching about pain management. International guidelines provided the basis for writing a series of articles addressing different facets of perioperative pain care. These were published in a special issue of Revista Mexicana de Anestesiologia, a national anesthesia journal.²⁹ Additionally, collaborators developed a website (www.painoutmexico.com) for use across Mexico. The website includes algorithms assisting providers in decision-making strategies of pain-relief for different types of surgery, instructional videos for programming patient-controlled analgesia pumps and for regional ultrasound-guided pain-relieving techniques. It also includes a section of news and announcements for continuing medical education in acute pain, and finally, a section for patient and family education. The Mexican College of Anesthesiology, the National Mexican Academy of Medicine and the Mexican Academy of Surgery supported the work of creating the website.

Programs for teaching providers were established in eight of the hospitals, and included anesthesiologists, at times surgeons, including residents and nurses. Patient leaflets were prepared in several hospitals. Procedure specific protocols were developed in six hospitals. An Acute Pain Service was established or work practices improved in three hospitals. In one hospital, further training was offered to the Acute Pain Service staff.

Qualitative assessment of the project’s effects on staff and patients. Before the project was initiated, staff were under the impression that their patients’ pain was well treated and that they could assess the adequacy of care by relying on the types of medications administered. However, the auditing of their own patients and obtaining the PROs, indicated to them that this was not so. The PROs revealed that their patients experienced severe pain and that they lacked information about pain treatment options. Providing information about pain treatment options was not a common practice, especially in public hospitals with a high patient volume. These sentiments were expressed repeatedly … ’we are not trained in managing postoperative pain … we think that we are prepared but this protocol helped me to understand that we are not. I think that all healthcare providers should be taught about pain management from the time they begin their training’. Another PI wrote, “I am deeply committed to the project and I will never return to my old practices!” Once protocols were developed and implemented, surgeons and nurses observed that patient outcomes improved. Patients were appreciative of the improved care they received. A PI wrote: “The patients feel very grateful with the whole process. Those who had surgery in the past and were currently undergoing surgery reported that they discerned a great deal of difference in pain management”. “We realized that patients undergoing thoracic surgery expect to have pain after surgery. We now teach them to identify severe pain and advise them to discuss this with the nurses and physicians, so that treatment options can be sought after”.

PIs repeated that the QI work is not a task that can be carried out by individuals and that creating multi-disciplinary teams of surgeons, anesthesiologists, and nurses is a perquisite for this type of work to succeed. They now recognize that introducing change is a difficult and lengthy process. Opposition to changing practice was met by individuals from all disciplines, surgeons, anesthetists and nurses. Strategies for addressing the opposition included: (i) offering to provide additional teaching; (ii) offering to review care given to specific patients and outcomes these patients reported; (iii) respecting the position and hoping that it may change with time.

Clinical Features of Implementing the Project

Changes at the Network Level

Figure 2A illustrates the relative frequencies of the PROs during the two project phases. For example, in phase one, the proportion of patients reporting ‘worst pain intensity of ≥ 6/10ʹ ranged from 42% to 73% between wards, with a median of 53%. In the second project phase, the corresponding proportion ranged from 0% to 64% between wards, with a median of 43%.

Figure 2 Distribution of the relative frequencies of the patient-reported outcomes are shown in (A) and for processes in (B). Each dot represents summarized data from one ward. Box plots filled in with gray, represent data for the first project phase and white plots represent data for the second phase.

Figure 3 (top panel) demonstrates that the primary outcome, ‘worst pain intensity ≥ 6/10ʹ, improved significantly between the project phases (RR: 0.69, 95%CI: 0.53 to 0.90, p = 0.006). The model predicted the risk of reporting ‘worst pain intensity ≥ 6/10ʹ for the first phase was 55%, which reduced to 38% in the second phase. Most of the PROs (including the composite score) improved significantly between the first and second phases. However, ‘nausea ≥ 4ʹ and “wish for more pain treatment” did not improve.

Figure 3 Changes due to the QI work at the network level. The marginal effects for project phase on the patient-reported outcomes are shown in the upper panel, shaded in light gray, and the process variables are portrayed in the lower panel, shaded in dark gray. Squares depict the relative risk regarding project phases obtained by regression modelling and the black horizontal lines indicate the corresponding 95 % confidence intervals. *p < 0.05, **p < 0.01, ***p < 0.001, p-values were not adjusted for multiple comparisons.

Figure 2B, illustrates the relative frequencies of the processes for the two project phases. For example, in phase 1, patient reporting receipt of information ranged from 28% to 68% between wards, with a median of 45%, and in the second phase, the range was 24–100% between wards, with a median of 85%. The regression analysis (Figure 3, lower panel), identified two significant overall effects for “treatment information” (RR =1.76, CI 1.57–1.98) and for assessment of pain (RR=1.13, CI 1.02–1.16). The model predicted that the risk or “opportunity” of receiving information about treatment options during the first phase of the project was 44% and this increased to 77% in the second phase. For the assessment of pain, the model predicted the risk or “opportunity” of having pain assessed during the first phase was 84%, and this increased to 95% in the second phase. A median of 20.9% (CI 16.7–25.1) patients at baseline and 20.0% (CI 19.5–30.5) patients after the intervention reported that they used or received a non-pharmacological intervention (Details about the types of interventions are listed in the Supplementary data file, S-3).

Detailed descriptive statistics and comprehensive information about the regression models are listed in the Supplementary data files S-4 and S-5.

Changes at the Ward Level

Findings from the single ward analysis for the PROs and processes are summarized in Figure 4.

Figure 4 Results of the single ward analysis and whether improvement took place and its effect size. Cells with a green background indicate improvement, whereas, red signifies worsening of the PRO or decreased implementation of the process in phase 2. The effect size for each item is written in each cell. + signifies potential ceiling effects, indicating that the process was implemented in >90% of cases in phase 1.

PROs are shown on the top panel. For example, the proportion of patients reporting worst pain ≥6/10, decreased in seven wards, there was no change in five wards and in one ward, the proportion increased. The effect size describing the extent of change, in each of the wards, was variable, ranging from −0.21 to 0.57.

Changes in the process (bottom panel) “receiving treatment information” improved in 9/13 of the wards. There was no change in four wards. Interestingly, the use of regional anesthesia increased in 7/13 wards between the two phases of the project. Effect sizes were medium to large in both cases.

Association Between PROs and Process

On the patient level (eg if an individual patient reported receiving information about pain treatment options), after correcting for multiple comparisons, only the associations between “treatment information” and four of the eight PROs (including the composite score) reached significance. On the ward level (eg if a ward demonstrates that a high proportion of regional anesthesia is carried out), after correcting for multiple comparisons, only “regional anesthesia” and “non-opioids” were associated with positive outcomes in more than one PRO. See Figure 5.

Figure 5 Associations between the PROs and processes. Cells in green depict significant regression coefficients indicating a favorable association between process variable and PRO (e.g. receiving information about treatment options is associated with a lower risk of reporting worst ≥ 6/10 NRS). Correspondingly, red cells depict significant regression coefficients indicating an unfavorable association between process variable and PRO (eg, receiving systemic opioids is associated with a higher risk of reporting nausea ≥ 4/10 NRS). Asterisks indicate significant associations after applying the Bonferroni-Holm correction for multiple comparisons and adjusted for p values of less than 0.05.

Discussion

Multi-disciplinary staff caring for surgical patients, in 13 wards, in 9 hospitals, completed a 2.5-year quality improvement study, in Mexico. In the overall network analysis, variability was observed in that six of the eight PRO measures and two of the eight processes improved significantly. On the individual ward level, considerable variability was detected as to whether the elements improved and their effect sizes. When improvements took place, they tended to cluster within certain wards and hospitals. The “treatment information” process was one that improved in the largest number of wards. The interventions that staff carried out were multi-faceted, and consisted of auditing, teaching staff about pain management, patients in some instances, developing, introducing or changing local pain management protocols. The low rate of missing data records indicates that patients were able to fill in the questionnaire for assessing the PROs and surveyors were able to collect the demographic and clinical data.

Particular Strengths of the Project

Detecting improvements in the PROs in the current study was striking. The intensity of pain decreased as did the interference items. Reduced interference of pain with activities such as moving in bed is particularly important for orthopedic patients, while less interference with taking a deep breath or coughing is generally relevant for patients undergoing abdominal or thoracic procedures. Lessening interference of pain with sleep is relevant for all surgical patients, independent of the type of surgery. Measuring PROs is particularly relevant and important when evaluating pain because pain is a highly individualized, subjective experience. As a general rule, QI studies prefer to evaluate processes rather than PROs. This could be because changing processes is more actionable and collecting process data is more straightforward, as it can be abstracted from patient files or administrative records.³⁰ When studies evaluate both types of measure, change tends to occur more readily for treatment processes rather than PROs. This might be related to most studies having limited time frames, which may not be long enough to observe changes in patient outcomes.^12,13

We suggest that carrying out the current project along the lines of a “network”, led by local clinicians, facilitated the exchange of information and developing local expertise in QI and pain management. Engaging local leaders increases the chance of sustained change by building local capacity, by addressing local organization- and provider-level implementation barriers, and by fostering local ownership of the change.³¹ In the current project, these features are illustrated by the publication of the pain management guidelines in a national anesthesia journal and developing the website for teaching and in plans for carrying out a follow-up project. An important contributing factor in the success of the project was the leadership of the network by the network leaders, and within each hospital, by the PIs. Strong team functioning, with effective communication and clarity about roles, seems to mediate improved evidence uptake in an interdisciplinary setting.³²

The commitment of the network leaders, PIs and teams working in each of the hospitals, was notable given that people volunteered their time for all aspects of the project in addition to their regular duties. Teams indicated that the project increased inter-professional collaboration in pain management between anesthetists and surgeons and nurses to a greater extent than before. They also indicated that the work was demanding. “It was a titanic task to change analgesic management practices that has been followed for years”.

Analysis of the data presented itself as a challenge. The patient sample was large and complex as patients underwent different surgical procedures, for a variety of surgical disciplines and received care involving diverse techniques and medications. However, we wished to use a common framework for evaluating management practices and the PROs, across all the wards. As such, we opted to employ “quality indicators”, which are measurable elements of practice performance for which there is evidence or consensus that they can be used to identify and assess gaps in practice and provide a direction for change.³³ The field of perioperative pain medicine lacks consented quality indicators.³⁴ Consequently, we proposed a framework for evaluating care that is “generic”, independent of surgical procedure, and for which a simpler format was used in an earlier study.²² Evaluating the indicators as to whether they are relevant, actionable and reliable,³⁵ will be work for future studies.

Nature of the Associations Between the Interventions and the Outcomes

The use of regional anesthesia and providing patients with information about pain treatment options were the only processes consistently associated with improved PROs. Pain measurement and administering a non-opioid were carried out in a high proportion of patients but had a little effect on the PROs. At this stage, we are unable to determine whether the level of associations between the PROs and processes we found, is due to these particular management processes being ineffective in providing pain relief in the clinical routine or the way the processes were implemented in the clinical routine or, possibly related to the methods used for the analysis.

Comparing Results with Findings of Other Publications

At the current stage of developing this change management program, it is not straightforward to suggest which factors and conditions contributed to the change, or lack of change, in the processes or PROs in the participating hospitals and wards. This is a recurring theme in reviews describing QI collaboratives and in studies where audit and feedback (A&F) is a principal component for driving change. As a rule, comprehensive details are generally missing as to which QI components were used, how intensively they were implemented and the extent that providers were engaged.^12,13 However, considering that QI collaboratives play a key role in contemporary strategies for accelerating improvement, but at best, they have only modest effects on outcomes, additional knowledge as to which components are effective in changing processes and/or outcomes and which are cost-effective, is crucial to determine the value of quality improvement collaboratives. A Cochrane review echoes similar conclusions regarding the effectiveness of A&F in change management projects.³⁶ A&F leads to small but potentially important improvements in professional practice. The authors caution that the use of the A&F has reached a stage, whereby it may no longer be ethically appropriate to continue directing human and financial resources towards trials using this methodology without addressing knowledge gaps as to when A&F works best and why and how to design reliable and effective A&F interventions.³⁷ Experts from diverse backgrounds outlined 15 recommendations for optimizing the effectiveness of A&F.³⁸ The best practices address: (i) methods for carrying out auditing, such as repeating data collection cycles over time; (ii) feedback that includes multi-modal presentations such as text, discussion and graphical materials, and (iii) the need to include treatment goals and action plans that are specific, measurable, achievable, relevant and aligned with personal and organizational priorities. Another approach was employed by Burton et al ³⁹ who interviewed participants in 12 quality improvement collaboratives involving 300 primary care practices, to identify components associated with success or failure of working within this framework. They found that people favor working as multi-disciplinary groups, learning from experts, peer-to-peer learning, and providing practical handouts that can be applied immediately in daily practice, such as clinical practice guidelines or patient screening questionnaires, in-person meetings lasting 4–6 hours, as opposed to conference calls.

Barriers to change are another feature of QI work. Though A&F is based on the premise that professionals will modify their performance when they receive feedback that their practice is not in line with a desired target,⁴⁰ in practice, healthcare professionals often face barriers that impede change from taking place. Barriers might be organizational, such as competing priorities of the organization; they can be due to a lack of leadership or skills or knowledge on how to interpret the feedback and to formulate appropriate improvement actions.⁴¹ Furthermore, even when healthcare professionals are aware of the barriers, they may not have the authority to address the barriers with their colleagues and thus, Gude et al⁴² recommend that interventions should be developed to identify and overcome barriers. In the current project, we did not address barriers in a formal way.

In the field of pain, Rawal ⁶ and Stamer et al ⁴³ suggest that A&F should be a regular component in the work of Acute Pain Services. However, the methodology for providing optimal A&F in the context of perioperative pain management has not been evaluated.

Focusing on processes and outcomes might be too simplistic a model as it does not take into account the multiple components within an organization that interact and might influence each other during the course of changing management of a clinical process. “Socio-technical” models might be used to design and evaluate A&F interventions. These models offer tools for evaluating the complex, multiple components addressed in A&F projects that continuously interact with and change each other, including people, teams, tasks, tools and technologies, underlying organizational conditions, and the surrounding context.⁴⁴ “Culture of the organization” may be another feature that should be evaluated in healthcare QI improvement studies in addition to the commonly used Structure-Process-Outcome model. Many definitions for “healthcare organizational culture” have been offered and they can be viewed as a metaphor for some of the softer, less visible, aspects of health service organizations and how these become manifest in patterns of care. Braithwaite et al ⁴⁵ found a consistently positive association between culture and outcomes across multiple studies, settings and countries.

Limitations and Efforts Made to Adjust the Limitations

In this study, we used one of the most commonly used pre–post-study designs, an uncontrolled before and after study. This is a study design often used where there are practical and ethical barriers to conducting randomized controlled trials.¹⁵ Though this is a relatively simple study design to conduct, it is superior to observational studies. The timeframe of the project was limited to 2 years. This relatively short time period aimed to keep staff engaged and to avoid losing staff who have gained experience with the project’s methodology, from moving to other positions, as part of their training. However, considering that QI programs are complex and multi-level, it is possible that changes may not have had time to evolve or those changes which were successful, may not have been sufficiently incorporated into the working culture so as to be sustainable over time.⁴⁶ Repeat data collection cycles and revisions of the intervention are one means for optimizing the effectiveness of A&F.⁴⁷ However, the framework of this study allowed for two cycles of data collection and one cycle for developing the intervention and implementing it. These issues may be addressed by a planned follow-up project, led by the same team. Feedback about the effects of the quality improvement interventions was provided to clinicians at the end of the project. We did not carry out an interim analysis which could have allowed for amendments and adjustments along the way. We are in the process of creating a “dashboard” that will offer clinicians an overview of the change in key quality indicators.⁴⁸ It should be dynamic and update at regular intervals, allowing clinicians to evaluate the effectiveness of the strategies they are using for change without delay.

Eighteen wards in 10 hospitals started the project. Fifteen wards took part in the second stage and staff in 13 were able to collect sufficient data in both phases, reflecting that this type of work is challenging and it is challenging to carry out over an extended period of time. Variability in processes and outcomes exists in many fields of medicine and gaining an understanding as to the causes for this is challenging.^49,50 In the current study, we demonstrate variability in outcomes and processes between wards and hospitals, however, we did not investigate potential underlying reasons for this. In a study evaluating PAIN OUT findings from 16,868 patients from several surgical disciplines, treated in 42 clinical centers, in 11 countries, the authors found that 94.3% of the total variance, related to satisfaction, occurred at the patient level, whereas, hospital, ward and country levels explained a small proportion of the variance, 3.7%, 1.1%, and 0.8%, respectively.⁵¹ Meissner et al ⁵² found considerable variability in pain-related outcomes for four commonly carried out surgical procedures as assessed in 138 hospitals. Outcomes were better in district hospitals compared to university or specialized medical centers. The authors attribute the differences to “soft factors”, such as higher levels of empathy, better communication, fewer fluctuations in staffing in the district hospitals. We are currently in the process of investigating this question in a large sample of wards and hospitals that participated in a pre–post study. A large and rich dataset was created during the course of this project. Yet, in this report, we analyzed only a small fraction of the data. We hope that the participants within this network will use the data to carry out additional studies and that it will be used by other collaborators in PAIN OUT, when carrying out multi-center, international evaluations of perioperative pain management. The teams in each hospital created their own quality improvement intervention. The advantage of this approach is that it allows each center to tailor the program to suit local conditions, yet, it is labor intensive. PAIN OUT is working to develop perioperative pain management intervention(s) that provides participants, across networks, with a standard structure that can then be tailored to the needs of each hospital. Lastly, patient assessments were carried out once, on POD1, and so we lack information about the process of pain resolution over time and whether this might have been improved due to the interventions. Assessments over multiple days are technically difficult to perform for reasons such as data anonymity, the need for additional manpower and patient fatigue. However, we intend to evaluate the feasibility of such methodology in a future project.

Conclusions

Many partners participated in this project, and so the question of its usefulness should be evaluated from a variety of perspectives. The network leaders found the project beneficial across the network to the extent they wish to carry out a follow-up study. Healthcare providers appreciated gaining increased awareness about the value of perioperative pain management, the need to improve it and that improvement work requires the collaboration of multi-disciplinary teams, across settings, namely, operating room, post-anesthesia care unit and ward. Obtaining results from the patient outcomes questionnaires made them aware that their patients experienced severe pain. However, not all providers found the project useful or considered that there was a need to change their practice. Overall, patients benefitted from better care and improved pain-related patient-reported outcomes. Patients who underwent surgery were aware of and appreciative of the improved care. We hope that future patients will benefit from those changes in care that will withstand the test of time.