Open Access Research Effectiveness and meaningful use of paediatric surgical safety checklists and their implementation strategies: a systematic review with narrative synthesis Janaka Lagoo,1 Steven R Lopushinsky,2 Alex B Haynes,1,3 Paul Bain,4 Helene Flageole,5 Erik D Skarsgard,6 Mary E Brindle1,2 To cite: Lagoo J, Lopushinsky SR, Haynes AB, et al. Effectiveness and meaningful use of paediatric surgical safety checklists and their implementation strategies: a systematic review with narrative synthesis. BMJ Open 2017;7:e016298. doi:10.1136/ bmjopen-2017-016298 ►► Prepublication history and additional material for this paper are available online. To view these files, please visit the journal online (http://d​ x.​doi.​ org/​10.​1136/​bmjopen-​2017-​ 016298). Received 5 February 2017 Revised 4 August 2017 Accepted 24 August 2017 For numbered affiliations see end of article. Correspondence to Dr Mary E Brindle; ​maryebrindle@​gmail.​com Abstract Objective  To examine the effectiveness and meaningful use of paediatric surgical safety checklists (SSCs) and their implementation strategies through a systematic review with narrative synthesis. Summary background data  Since the launch of the WHO SSC, checklists have been integrated into surgical systems worldwide. Information is sparse on how SSCs have been integrated into the paediatric surgical environment. Methods  A broad search strategy was created using Pubmed, Embase, CINAHL, Cochrane Central, Web of Science, Science Citation Index and Conference Proceedings Citation Index. Abstracts and full texts were screened independently, in duplicate for inclusion. Extracted study characteristic and outcomes generated themes explored through subgroup analyses and idea webbing. Results  1826 of 1921 studies were excluded after title and abstract review (kappa 0.77) and 47 after full-text review (kappa 0.86). 20 studies were of sufficient quality for narrative synthesis. Clinical outcomes were not affected by SSC introduction in studies without implementation strategies. A comprehensive SSC implementation strategy in developing countries demonstrated improved outcomes in high-risk surgeries. Narrative synthesis suggests that meaningful compliance is inconsistently measured and rarely achieved. Strategies involving feedback improved compliance. Stakeholder-developed implementation strategies, including team-based education, achieved greater acceptance. Three studies suggest that parental involvement in the SSC is valued by parents, nurses and physicians and may improve patient safety. Conclusions  A SSC implementation strategy focused on paediatric patients and their families can achieve high acceptability and good compliance. SSCs’ role in improving measures of paediatric surgical outcome is not well established, but they may be effective when used within a comprehensive implementation strategy especially for highrisk patients in low-resource settings. Introduction Checklists have been used in aviation, nuclear power and construction to manage complex Strengths and limitations of this study ►► There are few studies and no randomised controlled trials. ►► There was high variability in study quality, study designs and study populations. ►► There is insufficient support to make unambiguous and evidence-based recommendations regarding checklist implementation. ►► There are no other studies that quantitatively and qualitatively assess the critical factors for successful implementation of the surgical safety checklist (SSC) in the paediatric surgery population. ►► This systematic review highlights the role that parents play in a paediatric SSC. tasks associated with the risk of significant harm.1 Medicine is a similar high-stakes industry that has adopted the checklist. In 2007, a group of surgeons, anaesthesiologists and public health advocates working with WHO created a checklist to improve adherence to practices critical for safe surgery. The checklist was implemented in eight pilot hospitals internationally and reduced mortality at these sites significantly.2 Since its development, the surgical safety checklist (SSC) has been integrated into surgical systems worldwide.3–6 As of 2011, the WHO SSC had been adopted by more than 3900 hospitals in 122 countries.7 Paediatric surgical systems internationally have integrated SSCs into policy and accreditation standards.3 5 6 8 The SSC was developed primarily for adult patients, and the initial trial and most subsequent large-scale trials have been performed in adults.9 10 Evaluating the impact of this quality improvement tool in paediatric surgery requires a shift in perspective. The vast majority of children undergoing Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 1 Open Access surgery are healthy and experience low mortality rates when compared with their adult counterparts.11 12 Nevertheless, children still suffer from preventable surgical complications.13 What constitutes an acceptable complication rate is influenced by our knowledge of the rates accompanying adult surgery. Lower rates of complications in paediatric patients should not be reassuring. Paediatric patients have unique surgical risks. Physiologic challenges include the transitional circulation of the neonate and their high propensity to suffer from fluid losses and hypothermia during surgery. A child’s airway anatomy is different than that of adults. They have smaller blood volumes, immature immune systems and many other physiologic differences that challenge the surgical team. In addition to the patients’ physiologic differences, there are complex social issues that impact the structure and function of paediatric surgical systems. Communication strategies have evolved to anticipate a short preoperative period accompanied by significant stress when the patient enters the operating room prior to induction.13 The patient often has minimal understanding of the nature of their operation. Parents are called on to serve as advocates and decision-makers. As such, they have both a responsibility and an expectation to take an active role in the care of their paediatric charges. Information is sparse on how paediatric checklists are used, how they work and what they have achieved. These concerns are acknowledged by the Paediatric Surgical Chiefs of Canada (PSCC) who published their consensus opinion that a paediatric SSC is important for patient safety but also expressed their concern that information on the implementation and impact of paediatric SSCs is limited.8 The aim of our study is to investigate the effectiveness and meaningful use of paediatric SSCs and their implementation strategies within a systematic review of literature. Methods We have performed a comprehensive systematic review to synthesise published studies related to use of surgical checklists in children. Through this review, we collate evidence regarding the multiple measures of effectiveness of paediatric SSCs, to describe the strategies and attitudes related to implementation of paediatric SSCs, to evaluate the risk of bias in the evidence used to evaluate SSC use and to identify knowledge gaps in regard to the content or implementation of the SSC in children. This systematic review follows Preferred Reporting Items for Systematic Reviews and Meta-Analyses and the Meta-analysis of Observational Studies in Epidemiology standards.14 15 Clinical outcomes, process measures and attitudes are explored, and elements of implementation are synthesised using a narrative synthesis approach, with exploration of themes and contents through subgroup analyses. A priori, we established clinical outcomes, measures of compliance and assessment of barriers and facilitators as measures of interest. However, we also allowed for the identification of additional measures through the literature search. Our review presents a comprehensive synthesis of multiple study designs and objectives. We framed our question using the Population, Intervention, Controls, Outcome, Setting/Study (PICOS) type format. The population focus is paediatric patients, parents, caregivers and paediatric health systems. For the purposes of our work, we defined paediatric patients as those less than 18 years of age. The intervention is performance of a perioperative checklist defined as a list that exists in a physical and/ or electronic form used prior to an operation to ensure patient safety. We did not specify how the checklist was used. Controls were not required for inclusion. Any study involving an interventional procedure undertaken with a general anaesthetic was considered. All study types except case reports and non-original research were considered. Studies had to measure at least one outcome related to the checklist including clinical outcomes, compliance, attitudes and process measure outcomes. Elements of implementation were examined in included studies. The analysis takes the form of a narrative synthesis, exploring themes and contents through subgroup analyses. Idea webbing is used to further elucidate the components of implementation strategy that were successful and those that failed. Search strategy A broad search of electronic databases including Pubmed, Embase, CINAHL, Cochrane Central, Web of Science, Science Citation Index Expanded and Conference Proceedings Citation Index-Science was performed in partnership with a research librarian (PB) on 23 March 2016 (see online supplementary appendix 1). Inclusion and exclusion criteria were based on the PICOS elements described above. There was no language or date restriction. Data specific to paediatric patients needed to be available for separate analysis. Inclusion and exclusion criteria were agreed on a priori through team consensus. Non-English articles were translated by native speakers. Authors were contacted for missing data. Titles, abstracts and full texts were reviewed independently in duplicate (MB and JL) to generate the final list of articles for data abstraction. Discrepancies were resolved through consensus. Inter-rater agreement was measured. Citation searching was performed, searching both the references and citations of included papers. Conferences identified through the Conference Preceding Citation Index were screened in duplicate to identify relevant abstracts. The Data Extraction tool was generated through an iterative process. A pilot extraction created by the senior investigator (MB) was reviewed and modified by the research team (see online supplementary appendix 2). Data extraction was performed independently by three authors (MB, JL, SL) to ensure complete collection of 2 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 relevant information. During the synthesis phase, additional information was reabstracted to develop key themes. Quality assessment The quality of each study was assessed using risk-of-bias tools appropriate to study design including pretudies/ poststudies of compliance, cohort and cross-sectional studies as well as surveys. We developed this tool using the National Institutes of Health, Newcastle-Ottawa and CASP quality assessment tools (see online supplementary appendix 3). Using the tool as a guide, each study was given a rank of ‘good’, ‘fair’ or ‘poor’ by two reviewers. When two separate study elements were present (eg, clinical outcomes and compliance), both elements were evaluated separately. Overall quality assessment was obtained through discussion of ranking and final consensus of all three reviewers. We have summarised the risks of bias across studies describing limitations of methods, the quality and generalisability of the evidence as well as discrepant findings. Subgroup analysis was restricted to studies rated as ‘good’ or ‘fair’. Implementation information was abstracted if one of the subgroup analyses in a study was ‘fair’ or ‘good’. Synthesis A narrative synthesis was conducted as described by the Economic and Social Research Counsel Methods Programme16 using subgroup analyses in the form of tables and narrative description. Subgroup themes related Open Access to outcomes and meaningful checklist use were identified in an iterative fashion through examination of extracted data. Implementation strategies were explored in a descriptive fashion to identify the elements used within different strategies. A further subgroup analysis was used to describe educational components of implementation. We examined implementation strategies used across studies and the outcomes associated with various approaches. Relationships were explored visually using idea webbing to describe conceptual relationships between implementation strategies and outcomes.16 Results The electronic search identified 1921 citations. After screening, 84 papers were identified for full-text review with a Cohen’s kappa of 0.77. After full-text review, 37 articles were included with a kappa of 0.86. Disagreements were resolved through discussion between the two raters with a final decision by a third rater in situations of persistent disagreement. Thirteen studies were removed after data extraction and contacting the authors for data. Two additional papers were identified through citation searching. Twenty-six papers were included in the final systematic review (figure 1). The majority of studies were cohort studies with and without controls and cross-sectional studies. Ten studies had a predesign and postdesign related to implementation of the checklist or a strategy to improve checklist use (online supplementary table 1 describes included studies). Outcomes assessed Figure 1  PRISMA diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 3 Open Access within selected studies included attitudes, barriers/facilitators and effectiveness in terms of clinical and process measure outcomes. Study quality Study quality ranged from ‘poor’ to ‘good’. Six papers were judged as ‘poor’ for reasons including insufficient information provided about study design or an insufficient number of paediatric patients.17–19 ‘Good’ studies shared the following traits: strong survey quality, robust definitions of compliance and reliable measurements and robust definitions of intervention and reliable measurements.20 21 ‘Fair’ studies were those that had sufficient numbers of patients and sufficient information about study design but had minor flaws in design such as the use of survey tools that were not validated or the use of compliance measures prone to bias. For studies with two different elements evaluated (eg, compliance and survey), both elements were separately evaluated for quality and each element was considered for inclusion separately (online supplementary table 1 provides quality ranking scores) Parental involvement in the checklist Four studies explored the unique role of patients and parents in the performance of the SSC.8 19 22 23 Pires et al’s study involved a SSC designed specifically for patients and parents,23 while Corbally and Tierney, Skarsgard and Avansino et al explored the role of parents within a standard SSC.8 19 22 Pires et al reported that parents believed their involvement in a SSC improved patient safety and reduced their child’s anxiety.23 Parents in Corbally and Tierney’s study believed their participation improved patient safety and reassured them that the appropriate procedure would be performed. All parents felt that the parent role should be mandatory.22 Surgical team members surveyed in Corbally and Tierney’s and Avansino et al’s study agreed that parental involvement in the SSC improves patient safety.19 22 Skarsgard’s survey of Canadian chiefs of surgery indicated that these surgical leaders were more divided on their opinion of the role of parents with 6 of 12 surgical chiefs indicating that parental presence added little or no value.8 Study of checklist use and clinical outcomes The effectiveness of checklists at improving clinical outcomes was measured in three studies. These studies were population based and examined rates of morbidity and mortality before and after institution of an SSC. Two studies (Urbach et al and O’Leary et al) were performed in a similar population over a similar time frame leading to the likelihood of duplicate data.11 21 Urbach et al’s study demonstrated no change in mortality or infection rates after checklist introduction.11 Complication rates in both studies remained unaffected by checklist introduction.21 In addition, Urbach et al found that the checklist made no difference in length of stay or reoperation and O’Leary et al found that the checklist did not change the rate of emergency department visits or unplanned trips back to the operating room.11 21 A very different study by Jenkins et al demonstrated a dramatic decrease in mortality and morbidity in paediatric patients undergoing congenital heart surgery in a low-resource setting after implementation of a broad surgical safety implementation strategy including a checklist. In Jenkins et al’s study, the odds of mortality 1 year after checklist introduction was 0.71 of the baseline odds and it remained low at 0.76 in the second year. The decreased odds of infection were 0.65 that at baseline in the first year and 0.53 in the second year of the study.20 Urbach et al’s and O’Leary et al’s studies were retrospective, involving primarily low-risk surgeries with no standardised implementation process, while Jenkins et al’s prospective study examined high-risk surgeries in developing countries and employed a comprehensive implementation strategy, (table 1). Compliance with checklists Nine studies of ‘good’ or ‘fair’ quality evaluated compliance. The majority of studies (6/9) studied an intervention to improve compliance. Some studies (5/9) had baseline comparison populations either before implementation or immediately after implementation, while others had no comparison populations or used data from piloting the checklist prior to full implementation. Improved compliance was noted in all six studies involving an intervention to improve compliance. The authors of these studies reported that feedback, education and incorporating stakeholder solutions improved compliance. Two studies were notable for poor compliance. Levy et al’s study from 2012 noted little correlation between the excellent compliance reported by the hospital and the actual compliance with the SSC measured by a study auditor.24 Only 4 of 142 audits of SSCs in this study demonstrated completion of more than 50% of SSC elements. A survey of the surgical team members at Levy et al’s institution identified possible causes for poor compliance including a lack of understanding of both the roles and responsibilities of team members and the purpose and the timing of SSC. In addition, the authors ascribe poor compliance to the perceived failure of an adult checklist to address the needs of the paediatric population.24 In Ride et al’s study, agitation of the paediatric patient in the operating room was cited as the primary reason for skipping the checklist with coordinating the care team presenting an additional challenge (table 2).25 Implementation/operationalisation characteristics and educational strategies Eleven studies described the implementation and operationalisation of SSCs. Baseline data were collected in all studies. The majority of checklists were modified from the WHO SSC. Others were developed before the WHO SSC was published.26 27 Modifications of the WHO SSCs were generally based on feedback from pilot data or feedback from stakeholders. Stakeholder involvement was noted in eight studies. Putnam et al’s 2014 study noted that 4 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 Table 1  Impact of checklist introduction on mortality outcomes Study Jenkins et al20 Study number 15 049 operations Study context Study description Surgery for congenital heart disease 17 developing nations Prospective cohort study Complex prospective Intervention including checklist Preintervention cohort Reference year* 2010 or 2011 O’Leary et al21 28 772 operations Common paediatric surgery excluding neonatal surgery, cardiac and transplant ON, Canada Retrospective database (CIHI) analysis Prior and posthospital mandate of checklist Year prior to checklist adoption September 2008 to October 2009 Urbach et al11 15 495 operations All surgery ON, Canada Retrospective database (CIHI) analysis Prior and posthospital mandate of checklist 3-month period of time 3 months prior to checklist adoption (site specific) Postintervention cohort Year 1 2011 or 2012 Year 2 2012 only Year after all sites adopted checklist Sept 2010 to October 2011 3-month period of time 3 months after checklist adoption (site specific) Mortality change after Morbidity change after checklist checklist introduction introduction Mortality† Year 1 OR 0.71 (95% CI 0.62 to 0.81) Year 2 OR 0.76 (95% CI 0.69 to 0.83) Only one death recorded in the entire study (prechecklist) Mortality† OR 1.55 (95% CI 0.50 to 4.82)‡ Infection† Year 1 OR 0.65 (95% CI 0.58 to 0.73) Year 2 OR 0.53 (95% CI 0.48 to 0.58) Total complications† OR 1.01 (95% CI 0.90 to 1.14) Emergency visit† OR 1.06 (95% CI 0.93 to 1.21) Unplanned OR† OR 0.88 (95% CI 0.55 to 1.39) Total complications† OR 1.07 (95% CI 0.84 to 1.36) RR length of stay† 1.01 (95% CI 0.98 to 1.04) Readmission within 30 days† OR 1.12 (95% CI 0.92 to 1.37) *Enrolment was staggered; reference year was chosen as the first year prior to checklist introduction. †Risk adjusted, reference population prechecklist cohort. ‡Actual OR and CI obtained from study authors. If Entrolment was staggered, reference year was chosen as the first year prior to checklist introduction. CIHI, Canadian Institute for Health Information, RR, relative risk. Open Access 5 Open Access 6 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 Table 2  Studies of checklist compliance Article Study description Comparison population Method of measuring compliance Compliance measure Checklist compliance Comments Avansino et al19 Gottumukkala et al31 Khoshbin et al26 Levy et al24 Norton27 Norton and Rangel29 Study of intervention to improve Baseline at introduction compliance of checklist Compliance measured over 12 months after checklist introduction Monthly feedback of compliance provided to surgical team Audit Submitted paper checklists Compliance defined as any Improved compliance aspect of checklist completed Baseline compliance: Overall 88% Weekends 64% Weekdays 90% 12-month compliance: Overall 97% Weekends 100% Weekdays 94% Study of intervention to improve Baseline when compliance recording system in All cases videotaped place and evaluation Compliance measured over started 3 years through review of videos Feedback provided to team with ‘corrective action’ to improve compliance Audit 13% of videos of checklists performed Completion of each element of the checklist assessed by multiple raters Improved compliance (time series analysis) (crude values not available) Decreased variability in performance over time Compliance measured 1 year after introduction of a surgical ‘time out’ and 2 years after introduction of a preoperative ‘huddle’ None Audit Direct observation of ‘huddles’ and ‘time outs’ Compliance defined as any part of ‘huddle’ or ‘time out’ completed Completion measured for 4 elements of the huddle and 9 elements of the ‘time out’ ‘Huddles’: Compliance 64% Mean completion 3.2/4 ‘Time out’: Compliance 99.1% Mean completion 6.2/9 Compliance measured over 7-week period None Audit Direct observation of cases Compliance defined as completion of all preincision components of the surgical checklist Completion measured for the 13 elements in the checklist Hospital-reported compliance 100% True compliance (full checklist completed) 0% Time out compliance 97% Completion The average number of checklist items performed 4/13 4/142 had more than 50% of elements completed Study of intervention to improve Baseline at introduction compliance of checklist Universal Protocol introduced Education provided before implementation and feedback on effective use provided during audit Audit Direct observation and documentation by surgical team Study of intervention to improve Data from piloting compliance of checklist before Two separate pilots used implementation to develop and modify intervention Ongoing education during implementation Audit Direct observation of cases Compliance defined as time out and site verification completion Compliance defined as ‘usage of the checklist’ Improved compliance Site initialled: Observed 97%–100%. (increased from 88% first month) Documentation 97%–100% Time out compliance: Observed 97%–100% Documentation 97%–100% Improved compliance First pilot, overall compliance 80%–90% 6 months after introduction Sign in 90%–100% Sign out 95%–100% Feedback may increase compliance Lax definition of compliance Steady increase in compliance over 12 months Feedback and incorporating stakeholder solutions to compliance failures may improve compliance Possible selection bias and generalisability may be limited to interventional radiology but reliable measure of compliance Steady increase in compliance Meaningful completion of checklist can be marred by the perception of the checklist as a ‘task’ Unblinded audit Hospital-recorded compliance may not reflect implementation fidelity Meaningful completion of checklist can be marred by the perception of the checklist as a ‘task’ Factors commonly completed were patient identification and procedure Factors least completed were sterility confirmation, essential imaging, team member concerns and anticipated blood loss Education and feedback may maintain compliance Compliance does not always reflect meaningful completion Limited to Universal Protocol Addressing barriers identified through piloting the checklist and education may improve compliance The Hawthorne effect may lead to improved compliance Ongoing improvement in compliance Continued 7 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 Table 2  Continued  Article Ride et al25 Study description Compliance measured for 48 cases Comparison population None Montgomery et al30 Study of intervention to improve Baseline prior to compliance intervention Intervention to improve compliance using feedback provided to teams Putnam et al28 Study of intervention to improve Baseline prior to compliance interventions Staged strategy to improve compliance First stage: safety council and modification of checklist Second stage: workshops stakeholder audits/feedback Method of measuring compliance Audit Direct observation of cases Audit Direct observation of surgical pause Audit Direct observation of surgical pause Compliance measure Checklist compliance Measures include team Attendance for sign-in member attendance surgeon 10% Compliance defined as verbal Anaesthesiologist 95% verification of all checklist Verbal compliance: items Sign ins 95% Documentation of checklist Time outs 69% completion Sign outs 44% Correct use of checklist Documentation: Sign ins 79% Time outs 71% Sign outs 31% Correct use: Sign in and time out 46% Sign out 23% Compliance defined as performance of the surgical pause Completion measured for the 4 elements of the surgical pause Compliance improved Preintervention: Compliance 100% Completion of all 4 steps 51% Postintervention: Compliance 100% Completion of all 4 steps 77% Compliance defined as completion of entire checklist Completion measured for the 14 checkpoint elements. Compliance improved Baseline: Compliance 0% Completion 30% After first stage: Compliance 19% Completion 76% After second stage: Compliance 61% Completion 96% Comments The sign out is poorly done and correct use of the checklist is not common Correct use is not defined in this study Challenges to compliance reported by authors included patient agitation and difficulty coordinating members of the care team Feedback may improve completion Improvement of 4 elements: Introduction 61%–87% Identity and procedure 90%–100% Checklist performed 98%–100% Concerns voiced 71%–100% Feedback, education and incorporating stakeholder solutions to compliance failures may improve compliance Open Access Open Access stakeholder focus on paediatric-specific modifications to the SSC improved compliance beyond that obtained after a more general modification of the SSC for the parent institute (table 3).28 Checklists were implemented using a number of measures including marketing and raising awareness, institutional mandates, feedback as well as educational strategies. The use of feedback to improve compliance was described in a number of studies. Piloting was performed to provide feedback on checklist performance in 6 of the 11 studies. Norton and Rangel’s 2010 study addressed barriers noted during the piloting of the checklist to improve compliance.29 Similar success using feedback to improve compliance was described by Norton and Rangel29 and Montgomery et al.30 Identifying and addressing barriers based on feedback was particularly effective at improving compliance in the time series study by Gottumukkala et al.31 This study involved videotaping all checklist performances and regularly auditing a sample of these on an ongoing basis over years with continuous adaptation of the implementation process. The integration of these audits into regular practice resulted in an effective strategy of continuous quality improvement. Educational strategies within implementation models ranged from limited and focused educational interventions to expansive, sustained programme. Educational strategies were described in detail in seven studies (online supplementary table 2). Studies with a strong implementation plan demonstrated a positive impact of the checklist, while studies with limited implementation strategies demonstrated poor compliance, unchanged clinical outcomes and negative attitudes regarding institutional safety. Comprehensive strategies for patient safety with an integrated checklist appear particularly effective but the contribution of the checklist itself becomes difficult to discern. Conceptual but not necessarily causative relationships between implementation strategies and outcomes are explored through ideas webbing (figure 2). Attitudes related to the surgical checklist Eight studies explored attitudes related to checklist use and the impact of the SSC on the culture of surgical safety. These results were generally gleaned from surveys. Several positive attitudes related to the perceived value of the checklist were noted, but there were often conflicting views presented within studies. For example, Avansino et al’s study and Norton et al’s study from 2016 showed that there was strong agreement that the SSC is important for patient safety; however, only 59% of practitioners in Avansino et al’s study and 36% in Norton et al’s study felt that the SSC had actually identified patient safety issues.19 32 Positive attitudes to the checklist are highlighted in Norton’s publications from 2010 and 2016. Specifically, these studies conclude that nurses and physicians generally believe that checklists: improve patient safety, improve efficiency, prevent communication errors and work well in high-volume centres with significant distractions.29 32 Although the attitude towards surgical safety and the role of the SSC was generally positive, attitudes of nurses differed from physicians in some consistent ways. Nurses were more likely to perceive problems with communication and attitudes towards safety in their institution and more likely to view the checklist as playing an important role in improving patient safety (online supplementary table 3).19 26 32 Discussion This study is the first synthesis of peer-reviewed, published data concerning the impact of SSCs on paediatric patients. It is not a traditional systematic review in that it does not focus on a single area of inquiry. Our study tackles the scanty literature on paediatric SSCs by combining the broader research aims of a coping review with synthesis methods of a systematic review. Since its creation, the SSC has been integrated into paediatric surgical systems alongside adult surgical systems worldwide.3–6 Studies in adults have demonstrated that SSCs may be associated with reductions in postoperative morbidity when part of a strong implementation strategy that is tailored to the patient population.10 33 Paediatric patients and paediatric surgical systems have unique needs. Data on checklist implementation in the paediatric population remain limited. Understanding the utility, the barriers and facilitators of paediatric checklist use is crucial when developing policies and implementation strategies. Our review of the literature regarding SSCs in the paediatric population reveals several key themes regarding compliance, implementation and impact on outcomes. More than half of the studies in our review explored compliance.8 We identified challenges to compliance specific to the paediatric surgical setting. Many studies identified difficulty in performing a preinduction check in the operating room with an agitated child34 35 and the lack of buy-in without stakeholder-generated, paediatric-specific adaptations to the SSC.24 26 The definition of compliance in studies is inconsistent. Institutional compliance data often overestimate checklist use.24 If compliance is defined as completion of any aspect of the checklist, attaining high levels of compliance is not difficult.19 However, true compliance with the entire checklist is rarely obtained.24 Unblinded audits of compliance may be influenced by the Hawthorne effect.26 33 A further difficulty, and one that is much harder to evaluate, is determining meaningful compliance. An informal audit in one study identified that meaningful use did not always accompany good compliance.27 The concerns with the definition and measures of compliance were identified in a survey of PSCC who reflected on the dangers of using compliance as an administrative tool and the importance of developing strategies for meaningful use.8 Strategies to improve meaningful compliance in adult studies include stakeholder involvement in checklist development and SSC as well as implementation tailoring to meet the needs of the patient and the surgical team.36 37 8 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 9 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 Table 3  Implementation/operationalisation characteristics Study Avansino et al19 Checklist creation process (adoption or modification of existing checklist) Expected outcomes of implementation Stakeholder process engagement Use of baseline data Compliance with and paediatric needs to checklist revise WHO SSC Unclear Gottumukkala et al31 Institution-modified WHO SSC and Universal Protocol Adherence to Good Yes Practice Jenkins et al20 Site-specific Decreased mortality Yes stakeholder involvement Infection reduction to modify WHO SSC Improved perioperative practice Improved teamwork and communication Khoshbin et al26 Institution developed Non-WHO SSC Compliance with ‘huddles”and ‘timeouts’ Unclear Levy et al24 Montgomery et al30 Site-specific Compliance with stakeholder involvement checklist to modify WHO SSC WHO ‘surgical pause’ Compliance with Unclear if SSC modified checklist Yes Yes Key attributes of checklist implementation process Targeted Preimplementation ‘change education management’ Use of ‘pilot’ Use of CQI methods (eg, PDSA cycles or simple feedback) to improve implementation Outcomes Outcomes Documented sustainability Yes None Yes None Yes None Education Yes marketing and notifications: ►►Newsletters, bulletins, posters in operating room Use of feedback No educational strategy outlined Use of feedback No Comprehensive education strategy No Global aims: central organisation, support from NGOs Local implementation: ►►Local checklist modification and implementation ►►Translated materials Standardised ‘Practice Bundles’ and communication scripts Use of feedback No educational strategy outlined No Marketing and notifications: ►►Announcements at departmental meetings ►►Posters in operating rooms Limited educational strategy Notifications: ►►Posters in operating rooms No educational strategy outlined Use of feedback No Yes Compliance rates Improved compliance posted monthly during pilot Sustained compliance over 1 year Monthly audit of Improved compliance Continual videotaped procedures with process measures improvement Frontline teams use and decreased variability sustained across feedback to target multiple years lapses and create strategies to improve performance (PDSA)* Annual benchmarking Decreased mortality and decreased infections Sustained benefits in mortality and morbidity reduction at 2 years No No Feedback presented at surgical team department meetings Feedback used to create posters to improve compliance Compliance excellent with time outs and good with huddles Nurses view SSC as important for safety Physicians view SSC as a tool for increasing efficiency Poor compliance Poor implementation fidelity Improved Compliance Sustainability not assessed Sustainability not assessed Sustainability not assessed Continued Open Access Open Access 10 Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 Table 3  Continued  Study Norton27 Checklist creation process (adoption or modification of existing checklist) Expected outcomes of implementation Stakeholder process engagement Universal-Protocol- Compliance with based checklist created checklist by local champion and stakeholders Yes Norton and Rangel29 Site-specific Compliance with stakeholder involvement checklist to modify WHO SSC Process measure compliance Teamwork and communication measures Yes Putnam et al 201428 Use of baseline data Compliance with and site-specific checklist stakeholder involvement to modify WHO SSC Yes Putnam et al 201528 Institutional checklist based on WHO SSC modified by stakeholders Compliance with Yes appropriate antibiotic administration Wyrick et al 2015 39 Stakeholder (surgeon) involvement to modify WHO SSC Compliance with accurate SWC Yes Key attributes of checklist implementation process Targeted Preimplementation ‘change education management’ Use of ‘pilot’ Use of CQI methods (eg, PDSA cycles or simple feedback) to improve implementation Yes Yes Yes None Yes Comprehensive education strategy Marketing and notifications: ►►Marketing team ►►CEO letter to practitioners ►►Logo, flyers, posters, wearable button ►►Stickers on charts Mandated education No Education Marketing and notifications: ►►Marketing team ►►CEO letter to practitioners ►►Posters in operating rooms ►►Flyer in operating room sterile pack ►►Screen savers on computers with SSC reminder, newsletter Mandated checklist use Local leadership Use of feedback Yes Comprehensive education strategy Yes Marketing and notifications: ►►Distribution of web-based media ►►Posters in operating room Local leadership Use of feedback No educational strategy outlined Yes Marketing and notifications: ►►Notices on appropriate antibiotic use placed on anaesthetic carts ►►Emails sent to staff regarding guideline changes Computer order entry system to facilitate correct antibiotic use Education Marketing and notifications: ►►Bulletin boards near operating theatres Yes Results posted Feedback to staff monthly Feedback used to revise implementation strategy Feedback to staff Feedback from stakeholders and pilot testing used to revise implementation strategy Unclear No Outcomes Outcomes Excellent compliance throughout implementation Documented sustainability Sustained compliance over 14 months Improved compliance Continual improvement in compliance over 7 months Improved compliance None No improvement None in compliance with appropriate antibiotic use Improved accuracy of documented SWC Sustainability not assessed *Gottumukkala et al’s study did not specifically describe their implementation strategy as using the PDSA framework. However, given the fact that the elicited feedback from those using the checklists, collected and analysed this feedback and then used the feedback for checklist improvement, we have categorised this within the PDSA column. CEO, chief executive officer; CQI, continuous quality improvement; NGO, non-governmental oranisation; PDSA, Plan-Do-Study-Act; SSC, safe surgery checklist; SWC, surgical wound classification. Open Access Figure 2  Exploration of implementation approaches and outcomes using idea webbing. SSC, safe surgery checklist. Similar findings were apparent in our review of paediatric SSC studies. Adapting the implementation strategy to the culture and workflow of the paediatric surgical service, as described by Putnam et al, resulted in improved compliance.28 Feedback plays an important role in achieving compliance. A single episode of feedback can improve compliance.30 Frequent feedback over a longer duration demonstrates continued improvement of compliance to high levels sustained for the duration of the study.19 29 31 The incorporation of feedback into an ongoing strategy of continuous quality improvement is demonstrated by Gottumukkala et al’s study where institutional practice involved continuous auditing and iterative development of improvement strategies over multiple years. This entrenched strategy was associated with high compliance and minimal variability in performance.31 As in the adult surgery, meaningful checklist implementation enhances successful teamwork and communication in paediatric surgery. Some aspects, such as parental involvement, are unique to the paediatric population. Including the parent as an active participant in the SSC offers an opportunity for improved patient safety and patient/family satisfaction in the paediatric setting. Surveys of surgical team members directly involved in strategies to improve parental participation reflected a positive view of the parental role.19 22 In these studies, parental satisfaction was high and both parents and surgical team members believed that parental involvement improved patient safety.22 23 Corbally and Tierney points to the crucial role of the parent in situations where the members of the healthcare team change. Children are often removed from the process of consent and even mature children may not feel comfortable challenging a surgical plan. A parent, however, frequently assumes the role of protector and advocate and is more likely to challenge a different or unclear plan. Unlike other surgical team members, the parent of the patient has been consistently involved in the care of the child, has not suffered checklist fatigue and is not likely to regard participation in a surgical checklist as a routine activity. The parent assumes a unique role as a patient advocate in the preoperative setting and the development of a role for parental participation within a surgical checklist respects the philosophy of family-centred care and may Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 11 Open Access positively influence the culture of patient safety. Skarsgard’s study demonstrates that the views of the surgical team leadership on parental involvement in the SSC may be divided, and these divergent views might act as a barrier to developing a strategy of parental participation.8 Though there is still a paucity of literature exploring the role of parents within a checklist, current literature does highlight a parent’s importance in serving as patient advocates. Moreover, the involvement of parents in a SSC further promotes the principles of patient and family-centred care. The impact of SSC use on patient outcomes has been established in the adult literature. A few studies have started to shed light on the impact of the SSC on paediatric patient outcomes. Jenkins et al’s study demonstrates that a comprehensive strategy aimed at improving the quality and safety of surgery can improve patient outcomes.20 In this study, the SSC was just one of many interventions and the value of the tool cannot be discerned separately from the complex strategy of surgical safety in which it is implemented. The comprehensive undertaking over the 2 years of the study would require a substantial commitment by health systems hoping to replicate these findings. Although the gains possible within developing nations may not be achievable in other settings, similar high-risk vulnerable populations exist in high-income countries. O’Leary et al’s and Urbach et al’s studies remind us that a strategy that is primarily focused on mandating checklist use to improve outcomes in paediatric surgical patients is unlikely to provide much benefit. This is particularly true when dealing with low-risk patients in high-income settings.11 21 The disappointing findings of these database studies reflect those of adult studies that examine the impact of mandatory checklist use without a clearly defined implementation strategy. The checklist must be regarded as a tool within an integrated strategy for safety improvement with a strong focus on meaningful use. Our findings suggest that engagement of paediatric surgical stakeholders in the process of checklist adoption along with education of team members and parental participation in the SSC may improve team communication and meaningful checklist use. Evidence from our review suggests that SSC compliance may be improved using feedback. A comprehensive strategy such as that undertaken by Jenkins et al could improve clinical outcomes for children even in high-income countries. These results emphasise that the checklist may not achieve results when mandated as a stand-alone tick-box exercise, but it may achieve great success as part of a comprehensive quality improvement strategy. Our systematic review has some notable limitations and strengths. There are few studies and no randomised controlled trials that examine the performance of checklists in children. Only three studies specifically measured checklist impact on clinical outcomes. Reporting bias may have resulted in suppression of negative results. Some WHO SSC published studies that include paediatric patients within studies of patients of all ages might 12 not have been identified through our search strategy. Changes in outcomes seen in studies could be secondary to secular trends and compliance measures could be influenced by the Hawthorne effect. There is high variability in study quality, study designs and study populations. Thus, summarising and interpreting results is challenging. There is insufficient support to make unambiguous and evidence-based recommendations regarding checklist implementation. The strengths of this review are also notable. This study is the first synthesis that explores paediatric SSCs. We provide a broad quantitative and qualitative assessment of the critical factors for successful compliance with and implementation of the SSC in the paediatric surgery population. Our study is unique in its ability to highlight key attitudes held by members of the paediatric surgical team towards the checklist, its use in paediatric patients and its role in contributing to the culture of safety. Additionally, this systematic review highlights the role that parents play in a paediatric SSC and how their involvement can be optimised to improve patient safety. The findings of this review provide an evidence base that can guide paediatric surgical teams looking to implement strategies that will improve the positive impact of checklists. In conclusion, this review reinforces the findings from the adult literature that the SSC is not a standalone tool. Currently, the SSC often fails to respond to the unique needs of paediatric patients and the systems that care for them. Understanding contextual influences and addressing implementation fidelity are crucial in achieving meaningful outcomes.33 38 The results of this knowledge synthesis echo some of the key findings of the realist review of SSC implementation published by Gillespie and Marshall in 2015.37 In SSC implementation, similar strategies may achieve very different outcomes in different environments. SSCs can form a key part of initiatives to improve surgical safety for children. SSCs, however, require adaptation for the paediatric patient and environment both to improve patient safety and encourage acceptance. Parents may assume a unique role in the paediatric SSCs, which may improve the patient and family experience and ultimately improve surgical safety. Author affiliations 1Ariadne Labs: A joint center of the Harvard School of Public Health and Brigham and Women’s Hospital, Boston, Massachusetts, USA 2Section of Pediatric Surgery, University of Calgary, Calgary, Alberta, Canada 3Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts, USA 4Countway Library, Harvard Medical School, Boston, Massachusetts, USA 5Section of Pediatric Surgery, McMaster University, Hamilton, Ontario, Canada 6Division of Pediatric Surgery, University of British Columbia, Vancouver, British Columbia, Canada Twitter  @MaryEBrindle Contributors  All authors approve of the manuscript in its final form. JL aided in screening titles and abstracts and then full texts; she extracted data from final texts and helped in performing the narrative synthesis. She aided in interpretation of the data. JL created the first draft of the manuscript. SRL helped to create the Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 quality assessment tool and aided in data extraction and quality assessment of final papers. He also aided in interpretation of results and provided critical feedback for the paper. ABH helped establish inclusion and exclusion criteria and develop the data extraction tool. He provided critical feedback in review of the manuscript. PB created the search strategy for identification of articles for screening with input from MEB and JL. PB provided critical feedback in review of the manuscript. HF helped conceptualise the study. She provided critical feedback in review of the manuscript. EDS helped conceptualise the study, aided in framing of the results and provided critical feedback in review of the manuscript. MEB provided leadership in the project. She conceptualised the study, developed the methods and aided in screening, extraction and quality evaluation; with JL, she performed the narrative synthesis of the results and revised the manuscript. Funding  Funding for this project has been provided through the EQuIS research platform supported by the Brian and Brenda MacNeill Chair in Pediatric Surgery. Competing interests  None declared. Provenance and peer review  Not commissioned; externally peer reviewed. Data sharing statement  Additional data can be accessed via the Dryad data repository with doi:10.5061/dryad.k2v06. Open Access This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://​creativecommons.​org/​ licenses/​by-​nc/​4.​0/ © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted. References 1. Hales BM, Pronovost PJ. The checklist—a tool for error management and performance improvement. J Crit Care 2006;21:231–5. 2. Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 2009;360:491–9. 3. NHS England Patient Safety Domain. Standardise, educate, harmonise: comissioning the conditions for safer surgery. England: National Health Service, 2014. 4. The Joint Comission. Standards: Safe Surgery Checklist: The Joint Comission, 2012. http://www.​jointcommission.​org/​standards_​ information/​standards.​aspx 5. Accreditation Canada and the Canadian Patient Safety Institute. Making Care Safer. Ottawa, Canada: Accredication Canada and the Canaidan Patient Safety Institute, 2014. 6. Australian Commission on Safety and Quality in Health Care. Syndey, Australia. 2016 http://www.​safetyandquality.​gov.​au/​our-​work/​patient-​ identification/​patient-​procedure-​matching-​protocols/​surgical-​safety-​ checklist/ 7. Conley DM, Singer SJ, Edmondson L, et al. Effective surgical safety checklist implementation. J Am Coll Surg 2011;212:873–9. 8. Skarsgard ED. Recommendations for surgical safety checklist use in Canadian children's hospitals. Can J Surg 2016;59:161–6. 9. Neily J, Mills PD, Young-Xu Y, et al. Association between implementation of a medical team training program and surgical mortality. JAMA 2010;304:1693–700. 10. de Vries EN, Prins HA, Crolla RM, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med 2010;363:1928–37. 11. Urbach DR, Govindarajan A, Saskin R, et al. Introduction of surgical safety checklists in Ontario, Canada. N Engl J Med 2014;370:1029–38. 12. Bergs J, Hellings J, Cleemput I, et al. Systematic review and metaanalysis of the effect of the World Health Organization surgical safety checklist on postoperative complications. Br J Surg 2014;101:150–8. 13. Matlow AG, Baker GR, Flintoft V, et al. Adverse events among children in Canadian hospitals: the Canadian Paediatric Adverse Events Study. CMAJ 2012;184:E709–18. Open Access 14. Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:g7647. 15. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000;283:2008–12. 16. Popay J, Roberts H, Sowden A, et al. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version. 2006;1:b92. 17. Helmiö P, Blomgren K, Takala A, et al. Towards better patient safety: WHO Surgical Safety Checklist in otorhinolaryngology. Clinical Otolaryngology 2011;36:242–7. 18. Oak SN, Dave NM, Garasia MB, et al. Surgical checklist application and its impact on patient safety in pediatric surgery. J Postgrad Med 2015;61:92–4. 19. Avansino JR, Javid P, Katz C, et al. Implementation of a standardized procedural checklist in a children’s hospital. Am J Surg 2011;201:660–5. 20. Jenkins KJ, Castaneda AR, Cherian KM, et al. Reducing mortality and infections after congenital heart surgery in the developing world. Pediatrics 2014;134:E1422–E30. 21. O'Leary JD, Wijeysundera DN, Crawford MW. Effect of surgical safety checklists on pediatric surgical complications in Ontario. Cmaj 2016. 22. Corbally MT, Tierney E. Parental involvement in the preoperative surgical safety checklist is welcomed by both parents and staff. Int J Pediatr 2014;2014:1–3. 23. Pires MPdeO, Pedreira MdaLG, Peterlini MAS. Safe pediatric surgery: development and validation of preoperative interventions checklist. Rev Lat Am Enfermagem 2013;21:1080–7. 24. Levy SM, Senter CE, Hawkins RB, et al. Implementing a surgical checklist: more than checking a box. Surgery 2012;152:331–6. 25. Ride A, Bordes-Demolis M, Nouette-Gaulain K. Simplified opinion questionnaire using the patient safety checklist in a pediatric surgical unit. Annales Francaises D Anesthesie Et De Reanimation 2013;32:A214. 26. Khoshbin A, Lingard L, Wright JG. Evaluation of preoperative and perioperative operating room briefings at the Hospital for Sick Children. Can J Surg 2009;52:309–15. 27. Norton E. Implementing the Universal protocol hospital-wide. Aorn J 2007;85:1187–97. 28. Putnam LR, Levy SM, Sajid M, et al. Multifaceted interventions improve adherence to the surgical checklist. Surgery 2014;156:336–44. 29. Norton EK, Rangel SJ. Implementing a pediatric surgical safety checklist in the OR and beyond. Aorn J 2010;92:61–71. 30. Montgomery K, Thomson K, Khan I, et al. The implementation of the World Health Organisation's surgical pause at a pediatric surgical unit. British Journal of Surgery 2015:102–58. 31. Gottumukkala R, Street M, Fitzpatrick M, et al. Improving team performance during the preprocedure time-out in pediatric interventional radiology. Jt Comm J Qual Patient Saf 2012;38:387–94. 32. Norton EK, Singer SJ, Sparks W, et al. Operating room clinicians’ attitudes and perceptions of a pediatric surgical safety checklist at 1 institution. J Patient Saf 2016;12:44–50. 33. Haynes AB, Weiser TG, Berry WR, et al. Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention. BMJ Qual Saf 2011;20:102–7. 34. Lee S. The extended surgical time-out: does it improve quality and prevent wrong-site surgery? Perm J 2010;14:19–23. 35. Ride A, Bordes-Demolis M, Nouette-Gaulain K. Audit des pratiques de l’utilisation de la check-list sécurité du patient dans un bloc opératoire pédiatrique. Annales francaises d'anesthesie et de reanimation. 2013. 36. de Vries EN, Prins HA, Crolla RMPH, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med Overseas Ed 2010;363:1928–37. 37. Gillespie BM, Marshall A. Implementation of safety checklists in surgery: a realist synthesis of evidence. Implement Sci 2015;10:137. 38. Weiser TG, Haynes AB, Dziekan G, et al. Effect of a 19-item surgical safety checklist during urgent operations in a global patient population. Ann Surg 2010;251:976–80. 39. Wyrick DL, Smith SD, Dassinger MS. Implementation of the World Health Organization checklist and debriefing improves accuracy of surgical wound class documentation. Am J Surg 2015;210:1051–5. Lagoo J, et al. BMJ Open 2017;7:e016298. doi:10.1136/bmjopen-2017-016298 13