Unlocking the “black box” of practice improvement strategies to implement surgical safety checklists: a process evaluation

Background

Each year, an estimated 234 million surgeries are performed globally.¹ Of these, approximately seven million deaths occur,² half of which are believed to be preventable.^3,4 One strategy that has been advocated by patient safety experts worldwide to reduce patient morbidity and mortality is the implementation of surgical safety checklists (SSCs).^1,5 The main intent of checklists is to ensure that those “must do” clinical tasks are not forgotten and can be embedded into routine workflow patterns. In 2008, the Safe Surgery Saves Lives Group at the World Health Organization (WHO) launched a perioperative SSC¹ to minimize the risk of wrong site surgery. Following development, the WHO SSC was tested in eight diverse settings, including developing countries around the world.⁶ When the WHO SSC was used, there was a reduction of ~50% in surgical mortality, from 1.4% to 0.8% and a near 40% reduction in inpatient complications, from 11% to 7%.⁶ Since the publication of that landmark study, several meta-analyses suggest associations between checklist use and reductions in patient mortality, wound infection, pneumonia, blood loss, and postoperative complications.^7–9 Notwithstanding, improvements have been noted more broadly than just morbidity and mortality: for example, some researchers have found improvements in cost savings¹⁰ whereas others showed using an SSCenhanced communication, teamwork, and safety climate.^11–14 Taken together, these results suggest that safety improvements may be attributable to a change in work culture and improved team climate, although the mechanism of action to explain these changes remains less clear.

The challenges of implementation and sustained compliance have become apparent following widespread dissemination of the SSC across diverse countries and health care systems. Notably, implementation of checklists does not always lead to compliance. Full compliance to checklists is often difficult to achieve and largely depends on many contextual factors. Compliance differs between hospitals and staff members and between specific sections and individual items of the checklist.^{5–7,15–19} Variation in compliance has been noted: Borchard et al⁷ summarized 15 studies evaluating checklist compliance and found that overall compliance rates ranged from 12% to 100%. Other prospective studies based on administrative data have indicated that, while items were documented as being checked, clinical behaviors underlying those checks were not observed.^18,20–22 While researchers have evaluated outcomes relative to checklist compliance,^{5–7,15–19} there are currently no studies in this field that have evaluated strategies that support its implementation. As the main intent of using SSCs is to improve interdisciplinary communication among members of the surgical team, the implementation process should be aligned with this objective. Clearly, designing implementation interventions that promote behavioral engagement is central for such a checklist to fulfill its potential (or conversely avoid negative unintended impacts on safety). Thus, continuous evaluation of an implementation intervention should be part of its implementation and not an afterthought.

Complex interventions

Complex interventions are commonly described as interventions that contain several interactive components.²³ Components include techniques to facilitate behavior change that constitute the active ingredients of the interventions and the strategies used to deliver them.²⁴ Dimensions of complexity are characterized by the range of possible outcomes or their variability in populations and across contexts. Saliently, complexity is not characterized by the number of components within the intervention itself, rather it encompasses the interactions between components, the difficulty of behaviors by those delivering and receiving the intervention, and the degree of flexibility in tailoring that is allowed.²⁵ The lines between simple and complex interventions are blurred as complexity runs along a continuum; so in reality, few interventions are considered simple.²⁵ Consequently, the development and evaluation of complex interventions must be founded on a sound theoretical understanding to isolate the mechanisms that guide behavior and test the extent to which the “active ingredients” of the intervention magnify or diminish behavioral engagement.^25,26 A lack of impact may reflect implementation failure rather than effectiveness failure. In addition, the context in which the complex intervention is implemented may render strict fidelity inappropriate because the intervention may work better if adapted to the nuances of the context.^25,27 An important consideration while evaluating practical effectiveness centers on understanding the full range of an intervention’s effects and how these vary between participants and sites, over time, and the causes of those variations.^23,25

Materials and methods

Objective

The need to better understand how, when, and why particular strategies work prompted unlocking the “black box” in terms of checklist implementation. To this end, the objective was to undertake a process evaluation of the implementation of four knowledge translation (KT) strategies designed to support a behavior change intervention intended to promote better utilization of the WHO SSC. As part of this evaluation, a tentative logic model was generated to explain which KT strategies used to support the Pass the Baton (PTB) intervention worked well (or less well) in the operating rooms of a large tertiary referral facility in Queensland, Australia. This paper focuses on process outcomes, guided by the following questions:

1. To what extent were the KT strategies used to support the PTB intervention successfully and consistently implemented (ie, aspects of fidelity, dose, and coverage)?
2. To what extent were the KT strategies used to support the PTB intervention feasible and acceptable from participants’ perspectives (ie, what components were useful, why/why not)?
3. What are the processes underlying the change (or lack of change) (ie, evaluation of a logic model)?

Design

In this process evaluation, an observational design and triangulated data were used based on audit, fieldwork observations, and interviews. Process evaluations (as distinct from outcome evaluations) are important because they can clarify to what extent a complex intervention was delivered in a uniform way, whether the target population actually received the planned activities, what factors inhibited or promoted effectiveness, and what the participants’ actual experiences with the executed strategy were.^25,28 Process evaluations also provide information essential to understanding and validating theory-informed practice improvement strategies.²⁵ Analysis of process and outcome data allows evaluations to explore associations between strategy delivery and receipt, and outcomes on effectiveness.²⁹ In this process evaluation, a logic model was developed to help explain causal mechanisms of the implementation strategies that were used to support PTB.

Logic model development

Arising from process evaluation, logic models are based on the need to explore and make explicit links between inputs, activities, actions, and outcomes and thus offer a conceptual model about how the intervention might work.^28,29 Logic models may be used to help identify under what environmental conditions the implementation would work best and/or to evaluate an existing strategy as it is being implemented.²⁵ The purpose of this process evaluation was to focus on the short-term outcomes and mechanisms of action that gave rise to those outcomes. In the current study, the logic model presented in Figure 1 framed the evaluation questions, focusing on the KT strategies and their mechanism of action.

Figure 1 Checklist implementation logic model. Abbreviation: PTB, Pass the Baton.

Intervention context

The setting for this process evaluation was a 750 bed tertiary hospital with 22 commissioned operating rooms in Queensland, Australia. The aim of the PTB intervention was to improve checklist use and therefore the communication of clinical information as a team.

The theoretical underpinnings of the PTB intervention were based on two sources: 1) literature on checklist implementation,²⁷ and 2) data derived from an analysis of the barriers and enablers of checklist implementation in surgery.^15,30,31 The first source, a realist review,²⁷ was undertaken to explain when and how implementation of SSC KT strategies worked, or did not work well, and why. Key findings revealed that using surgical checklists was discipline-specific and that greater uptake was achieved when clinician stakeholders were involved in tailoring and evaluating implementation.

The second source, a prospective study, included a barrier analysis based on 33 interviews with 70 stakeholder participants.^15,31 The Theoretical Domains Framework³⁰ informed stakeholder interview questions to identify modifiable barriers and enablers that influenced the participation of surgical teams in all safety checks. The top four barriers were workflow, a lack of knowledge about timing/content of checks, a lack of clinical leadership, and negative/skeptical attitudes. Stakeholders considered that workflow had the greatest impact on their ability to be present and participate in all phases of the SSC. Consequently, workflow was systematically mapped with stakeholders.³¹ Findings indicated that the organization of work activities and the need to multitask during time-critical periods prevented team members from participating in various safety checks. Pragmatically, this meant that current checking processes were not feasible and needed to be tailored to incorporate workflow needs.³¹

Ethics review

This study was reviewed by the ethics committees of hospital (HREC/13/QGC/154) and the university (NRS/06/14/HREC). Participants gave informed consent for observations and interviews and were assured that participation was voluntary and their decision to decline participation would not prejudice their professional or employment status. Participants were advised of their rights to discontinue participation anytime throughout the data collection period.

Data collection and analysis

Data were collected over 3 months, during 2015–2016. To evaluate consistency (ie, fidelity, dose, coverage) of implementation, 35 surgical teams and 30 education sessions were audited using a standardized data collection tool specifically developed for the evaluation. In assessing feasibility and acceptability of implementation, semi-structured interviews were used. Interviews and audits were conducted by two members of the research team. Observations of surgical teams and interview participants were sampled conveniently. An interview guide was developed, and the interviews were intended to enrich the observational findings and provide supporting data on feasibility and acceptability. Implementation fidelity, dose, and coverage were captured through observational audit. The number of posters displayed in the department, the number of staff who received lanyards and pens, the number of education sessions delivered (by whom, number of attendees, and duration), and the presence/number of change champions were documented.

Individual or group interviews with participating interdisciplinary team members lasted between 10 and 30 min. Interview questions included: Were the implementation strategies delivered? If so, how were they delivered? Were they delivered the way they were intended to be delivered? How did you find using the implementation strategies? In all, 15 semi-structured interviews were conducted, including five group interviews, with a total of 96 participants. Of these, 56 were nurses, and 40 were physicians from anesthetics and surgery.

Observational data were analyzed using the Statistical Package for Social Sciences (SPSS V 22; IBM Corp, Armonk, NY, USA). Data entry was checked for accuracy and analyzed descriptively using absolute (n) and relative (%) frequencies. For interviews, a deductive content analysis approach was used as described by Elo and Kyngash³² to prepare, organize, and report findings relative to feasibility and acceptability of the implementation of the KT strategies. As part of the analytic process, words or phrases were coded and categorized as either barriers or enablers. The decision to classify codes as either a barrier or an enabler was relative to its perceived impact on participants’ clinical practice.

Underpinning assumptions and intervention

The following underpinning assumptions were supported by behavior change theories^33–35 and checklist literature:^16,27,36

1. Staff value the SSC as a safety tool for enabling team-based communication.
2. Staff are accessible and willing to participate in the KT strategies and the PTB intervention.
3. Implementation of the PTB intervention will result in improvements in team communication and work processes.

The KT strategies were “cocreated” and implemented with stakeholder input, with the ultimate aim of supporting the PTB intervention. “Cocreation” involves developing a shared body of knowledge to close the gap between evidence and practice in partnership with the end users – clinicians.^37,38 Table 1 details each of the four strategies and their processes and the proposed mechanisms of action.

Table 1 KT strategies, targeted barriers, and processes that support the implementation of PTB Abbreviations: PTB, Pass the Baton; KT, knowledge translation; TDF, Theoretical Domains Framework.

As workflow was a major barrier, it was important that the PTB intervention itself (as distinct from the KT strategies that supported it) addressed this barrier. The most problematic phases of the SSC where staff participation/usage was at its lowest were during the “sign in” (52.1%) and “sign out” (0.0%) phases.^15,31 When checks were performed during “sign in,” they were not verbally confirmed with others in the surgical team because they were not present or were performing other priority tasks during that time. “Sign out” was not done at all as a team despite that most members were still present during wound closure.³¹ Enabling staff to better utilize the SSC required tailoring the SSC in relation to: 1) the person delegated to performing/leading the checks, and 2) the timing and confirmation of these checks with others. Stakeholders agreed collectively that nurses were pivotal to facilitating the checks and acting as conduits of information to others. The PTB intervention is depicted in Figure 2 and described below:

• To increase team member participation in “sign in” anesthetic nurses initiated “sign in” and verbally confirmed relevant information to other members of the team (in the operating room) at an appropriate time.
• To facilitate team member participation in “sign out” instrument nurses led this phase, and verbally confirmed each checklist item with other team members during wound closure.

Figure 2 Pass the Baton intervention, mapping flow of communications using the checklist. Abbreviations: A/B, antibiotics; PTB, Pass the Baton; AN, anesthetic nurse; ANA, anesthetist; SS, scrub/scout nurse; SUR, surgeon; VTE, venous thromboembolism.

Results

Selection of KT strategies, that is, change champions/leaders, education, audit and feedback, and prompts and reminders, was data driven.^15,31 Results of the observational audit indicated moderate consistency in implementation. Prior to implementation, staff from surgery, anesthetics, and nursing were notified via email on two occasions. In terms of overall coverage, 160/220 (72.7%) staff including nurses, anesthetists, and surgeons knew generally about the implementation strategies. Introduction of each KT strategy was staggered over four 1-week periods.

Week 1 – Opinion leaders/change champions

The strategy was to identify opinion leaders from surgery, anesthetics, and nursing. The greatest buy in and engagement came from the nurse leaders in education. The Perioperative Educator identified four nurses working across surgery and anesthetics who assumed the role of change champions to lead implementation. Despite attempts to identify change champions in surgery and anesthetics, very little engagement occurred. Implementation of opinion leaders and change champions was perceived as useful by 81/96 (84.4%) interview participants.

Weeks 2–4 – Information sessions

Brief education sessions combining audit and feedback were implemented, and the results from process mapping workflow were presented to staff. Process map results, detailing workflow issues, resonated well and increased buy in, particularly from nursing staff. Of the 30 education sessions audited, eight (26.6%) involved PowerPoint presentations that were delivered by a designated change champion and lasted between 20 and 40 min. Of these, 7/8 sessions targeted nursing staff with 20–40 nurses in attendance at each session. The remaining education session was attended by 35 anesthetists (consultants and registrars) in the anesthetic department and was delivered by one of the study investigators. The other 22 (73.3%) informal sessions were delivered ad hoc by two nurses who were designated change champions. These sessions targeted any nursing or medical staff present, lasted between 5 and 10 min, and occurred during case setup periods, between cases, and during downtime periods. Assembling surgeons en masse was unfeasible because of competing work commitments that spanned across several different hospitals in the health services district. As such, surgeon consultants and registrars were targeted during the ad hoc education sessions that occurred serendipitously. Of the 96 participants interviewed, 91/96 (94.8%) believed that receiving some form of education was useful, particularly including audit and feedback as part of the education strategy.

Week 4 – Prompts and reminders

A total of 14 A3-sized posters were visible throughout the 22 theatre complexes. Ten posters were displayed in department corridors, whereas four others were located on the education board, reception area, recovery room, and staff tearoom. During audit, at least three members in 16/35 (45.7%) teams were wearing lanyards with the PTB logo. While 200 pens (with the PTB logo) were distributed to nursing and medical staff, they were not visible during the audit period. In relation to reminders, 68/96 (70.8%) participants believed that the posters and lanyards were useful prompts.

Table 2 displays the breakdown of findings relative to the feasibility and acceptability of implementation. A notable barrier to feasibility was physician engagement. An obstacle to acceptability was participants’ uncertainty about how implementation strategies could/would improve clinical practice.

Table 2 Verbatim based on 15 informal qualitative interviews* with 96 participants Note: *Data based on individual and group interviews. Interview questions were based on the Theoretical Domains Framework. Abbreviations: RN, registered nurse; ANA, anesthetist; SUR, surgeon; PTB, Pass the Baton.

Discussion

This is the first study of its kind in this area to evaluate the implementation of KT strategies designed to support use of the SSC. Clinical stakeholders were engaged a priori to undertake an evaluation of the implementation, and a logic model was used to inform the process component of the evaluation. The results of this evaluation suggest that implementation of KT strategies was somewhat successful relative to fidelity, feasibility, and acceptability. The four KT strategies employed do not encompass the full gamut of potential KT interventions, and others may need to be considered to ensure sustainability. However, the development of the strategies was both theoretically informed and data driven. Most practice improvement interventions that rely on behavior change are implemented using pragmatic approaches, in response to emergent local needs rather than first identifying barriers and enablers to implementation within a theoretical framework. The approach to implementing these KT strategies, while responsive to the ever-changing contextual demands of the surgical environment, was informed by theory^27,38 and empirical data, triangulated from a barrier analysis.^15,31

Mechanism of impact

In designing this process evaluation, Pawson²⁶ context–mechanism–outcome was applied to articulate the probable causal mechanisms and the effects that context (ie, anything external to the intervention that acts as barrier or enabler) had on the success (or failure) of implementation. In unlocking the “black box” of implementation, “mechanism” yields an understanding of the workings of the intervention that bring about the effects or changes in behaviors.³⁹ Seldom can any observed change be attributed to one mechanism (ie, cause). Thus, attribution of any one of the four KT strategies to desired behavioral outcomes might be contestable as there are usually several mechanisms of action operating simultaneously that could explain the success of implementation. Understanding how participants interact with a complex intervention is critical to understanding how they work.²⁵ The impact of context on participation in implementation was evident in varying levels of interaction and engagement: contextual factors such as team culture (ie, history, interpersonal relationships) and the characteristics of participants (ie, professional role, level of knowledge/skills/experience, beliefs/attitudes) likely influenced the degree to which participants interacted with the implementation strategies. Indeed, the biggest barrier to the implementation was physician engagement. Physician interaction with implementation was generally limited but when senior nurses initiated and led implementation, their support and participation increased.

During the evaluation, nurse leaders reported variable uptake of the KT strategies among some surgical specialties, depending on the context and the individuals involved. Engagement of nursing leaders was pivotal to implementation success. A key strategy targeting participant engagement with PTB was the use of opinion leaders and change champions. During implementation, opinion leaders led both formally and informally through their attitudes and overt behavior and were at the center of interpersonal communication networks within the department. A systematic review of 12 studies on the effectiveness of opinion leaders’ strategies concluded that their implementation led to improvements in compliance, with desired practice ranging from an absolute improvement of 25% to a decline of 6% with an overall median improvement of 10%.⁴⁰ In this evaluation, the education/information sessions were well attended, and stakeholder participation was interactive. Educational sessions offered to large numbers of learners are common, but some evidence suggests that this type of KT strategy produces little, if any, behavior change. Yet, other studies have indicated that sessions where there is increased interactivity, variation in teaching methods, and linked to assessment are likely to be more effective in improving clinical performance.⁴¹ The education sessions implemented in this study focused on strategies designed to optimize checklist use taking into account workflow issues. Audit and feedback on performance using the checklist were included, and overall the feedback from stakeholders was positive. The underlying mechanisms of action with audit and feedback are increasing motivation by showing the gap between actual and desired performance.⁴² A Cochrane Review of 140 randomized controlled trials concluded that audit and feedback lead to small but potentially important improvements in clinical practice.⁴³ However, effectiveness depends on baseline performance and how feedback is provided.⁴³ Delivery of feedback varies according to format (ie, verbal, paper, electronic), frequency and duration, content, and use of various resources.⁴² During implementation of PTB, reminders such as posters, lanyards, and pens were used as prompts. While reminders may become ineffective over time if they are not modified,⁴² they enable self-regulation and reduce the cognitive load for clinicians, thus promoting habit formation for the behavior. However, as habit formation depends on the repetition of a behavior, it may be that other behavior change techniques focused on self-regulation and sustaining motivation, like goal setting, self-monitoring, learning associations, behavior substitution, and repetition are needed.⁴⁴ Therefore, the role of habit formation as a key mechanism involved in the development and maintenance of using the checklist is worthy of further investigation.

Limitations and strengths

One of the key strengths of this process evaluation was that it was cocreated with stakeholders across nursing, surgery, anesthetics, management, and education. Through its participatory design, stakeholders at all levels within the organization were actively involved in specifying the KT strategies designed to support the implementation of the PTB intervention. Hence, it is more likely that the intervention will be sustained over time. Logic model methodology was incorporated to inform the evaluation, but while logic models represent intention, they do not reflect reality.²⁸ Nevertheless, in developing the model, stakeholders and evaluators are forced to focus on and be accountable for what matters – formative and summative outcomes. Furthermore, by adopting a theoretically grounded intervention which is operationally supported by logic model methodology, probable causal mechanisms that bring about any observed effects or changes in clinical behavior could be identified. Fidelity in the implementation of the KT strategies herein was reasonable – however, external validity may be limited because day-to-day implementation is needed to respond to the ever-changing clinical demands of the surgical environment. Consequently, participants adapted implementation to local conditions on some occasions during the 3 month roll out period. These minor variances in fidelity were documented as part of the process evaluation.

Conclusion

Results of this process evaluation suggest reasonable success in implementation of four KT strategies to embed the PTB intervention but refinement and/or addition of other strategies is necessary given the limited engagement of some stakeholder groups. Outcome (ie, summative) evaluation of the medium- and long-term impacts of the PTB intervention itself is underway. After further refinement and tailoring, PTB will be implemented on a larger scale, across multiple hospital sites. Stakeholders and evaluators should continually reflect on and modify the implementation of any behavior change intervention and the KT strategies that support it in response to theory, context, practice, and policy changes that ensue over time. While a body of work has informed this implementation effort, this evaluation is exploratory and further research is needed to move toward undertaking a definitive evaluation.

Acknowledgments

BMG was supported by a National Health and Medical Research Council (Australia) Translation into Practice (TRIP) Fellowship and the National Centre for Research Excellence in Nursing. The study was funded through Griffith University and Gold Coast University Hospital Foundation Collaborative and Centre for Health Practice Innovation grants.

Author contributions

All authors contributed toward data analysis, drafting and critically revising the paper and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		World Health Organization. Implementation of the Surgical Safety Checklist. Geneva: World Health Organization; 2008: 1–28.
2.		Weiser T, Regenbogen S, Thompson K, et al. An estimation of the global volume of surgery: a modelling strategy based on available data. Lancet. 2008;372(9633):139–144.
3.		Kable A, Gibberd R, Spigelman A. Adverse events in surgical patients in Australia. Int J Qual Health Care. 2002;14(4):269–276.
4.		Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ. 2001;322(7285):517–519.
5.		de Vries EN, Hollmann MW, Smorenburg S, Gouma DJ, Boermeester MA. Development and validation of the SURgical PAtient Safety System (SURPASS) checklist. Qual Saf Health Care. 2009;18(2):121–126.
6.		Haynes AB, Weiser TG, Berry WR, et al. A Surgical Safety Checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360(5):491–499.
7.		Borchard A, Schwappach D, Barbir A, Bezzola P. A systematic review of the effectiveness, compliance, and critical factors for implementation of safety checklists in surgery. Ann Surg. 2012;256(6):925–933.
8.		Bergs J, Hellings J, Cleemput I, et al. Systematic review and meta-analysis of the effect of the World Health Organization surgical safety checklist on postoperative complications. Br J Surg. 2014;101(3):150–158.
9.		Gillespie B, Chaboyer W, Thalib L, Fairweather N, Slater K. Effect of using a safety checklist in surgery on patient complications: a systematic review and meta-analysis. Anaesthesiology. 2014;120(6):1380–1389.
10.		Semel M, Resch S, Haynes A, et al. Adopting a surgical safety checklist could save money and improve the quality of care in US hospitals. Health Aff (Millwood). 2010;29(9):1593–1599.
11.		Sewell M, Adebibe M, Jayakumar P, et al. Use of the WHO surgical safety checklist in trauma and orthopaedic patients. Int Orthop. 2011;35(6):897–901.
12.		Kearns J, Uppal V, Bonner J, Robertson J, Daniel M, McGrady E. The introduction of a surgical safety checklist in a tertiary referral obstetric centre. BMJ Qual Saf. 2011;20(9):818–822.
13.		Yuan CT, Walsh D, Tomarken JL, Rachelle A, Shakpeh J, Bradley EH. Incorporating the World Health Organization Surgical Safety Checklist into practice at two hospitals in Liberia. Jt Comm J Qual Patient Saf. 2012;38(6):254–260.
14.		Kawano T, Taniwaki M, Ogata K, et al. Improvement of teamwork and safety climate following implementation of the WHO surgical safety checklist at a university hospital in Japan. J Anaesthesiology. 2014;28(3):467–470.
15.		Gillespie BM, Withers T, Lavin J, Gardiner T, Marshall AP. Factors that drive team participation in surgical safety checks: a prospective study. Patient Saf Surg. 2016;10:3.
16.		Fourcade A, Blache JL, Grenier C, Bourgain JL, Minvielle E. Barriers to staff adoption of a surgical safety checklist. BMJ Qual Saf. 2012;21(3):191–197.
17.		García-París J, Coheña-Jiménez M, Montaño-Jiménez P, Córdoba-Fernández A. Implementation of the WHO “Safe Surgery Saves Lives” checklist in a podiatric surgery unit in Spain: a single-center retrospective observational study. Patient Saf Surg. 2015;9:29.
18.		Levy S, Senter C, Hawkins R, et al. Implementing a surgical checklist: more than checking a box. Surgery. 2012;152(3):331–336.
19.		Sparkes D, Rylah B. The World Health Organization Surgical Safety Checklist. Br J Hosp Med (Lond). 2010;71(5):276–280.
20.		Cullati S, Licker MJ, Francis P, et al. Implementation of the surgical safety checklist in Switzerland and perceptions of its benefits: cross-sectional survey. PLoS One. 2014;9(7):e101915.
21.		Vats A, Vincent C, Nagpal K, Davies R, Darzi A, Moorthy K. Practical challenges of introducing WHO surgical checklist: UK pilot experience. BMJ. 2010;340:b5433.
22.		Biffl W, Gallagher A, Pieracci F, Berumen C. Suboptimal compliance with surgical safety checklists in Colorado: a prospective observational study reveals differences between surgical specialties. Patient Saf Surg. 2015;9:5.
23.		Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337(a1655):969–983.
24.		Michie S, Abraham C, Eccles M, Francis J, Hardeman W, Johnston M. Strengthening evaluation and implementation by specifying components of behaviour change interventions: a study protocol. Implement Sci. 2011;6:10.
25.		Moore G, Audrey S, Barker M, et al. Process Evaluation of Complex Interventions UK Medical Research Council (MRC) Guidance. Southhampton, UK: MRC Population Health Sciences Research Network; 2014: 132.
26.		Pawson R. Evidence-Based Policy: A Realist Perspective. London: Sage; 2006.
27.		Gillespie BM, Marshall A. Implementation of safety checklists in surgery: a realist synthesis of evidence. Implement Sci. 2015;10:137.
28.		Taylor-Powell E, Henert E. Developing a Logic Model: Teaching and Training Guide. Madison (WI): University of Wisconsin; 2008.
29.		W K Kellog Foundation. Logic Model Development Guidance. Missouri: W K Kellogg Foundation General; 2006.
30.		Michie S, Johnson M, Abraham C, et al. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.
31.		Gillespie BM, Marshall AP, Gardiner T, Lavin J, Withers T. The impact of workflow on the use of the Surgical Safety Checklist: a qualitative study. ANZ J Surg. Epub 2016 Jan 7.
32.		Elo S, Kyngash H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107–115.
33.		Michie S, Fixsen D, Grimshaw J, Eccles M. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci. 2009;4:40.
34.		Michie S, van Stralen M, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(42):1–12.
35.		Crane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.
36.		Russ S, Sevdalis N, Moorthy K, et al. A qualitative evaluation of the barriers and facilitators toward implementation of the WHO Surgical Safety Checklist across hospitals in England lessons from the “Surgical Checklist Implementation Project”. Ann Surg. 2015;261(1):81–91.
37.		Peppler, D. Stepping sideways to move forward: Closing the science-practice gap. Can Psychol. 2016;57(1):44–50.
38.		Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.
39.		Pawson R, Tilley N. Realistic Evaluation. London: Sage; 1997.
40.		Flodgren G, Parmelli E, Doumit G, et al. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2011;(8):CD000125.
41.		Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of continuing medical education. Evid Rep Technol Assess. 2007;149:1–69.
42.		Wensing M, Boach M, Grol R. Developing and implementing knowledge translation interventions. CMAJ. 2010;182(2);E85–E88.
43.		Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2012;(6):CD000259.
44.		Allom V, Mullan B, Cowie E, Hamilton K. Physical activity and transitioning to college: the importance of intentions and habits. Am J Health Behav. 2016;40(2):280–290.