Open Access

Research

The impact of feedback of intraoperative technical performance in surgery: a systematic review
Abhishek Trehan,1 Ashton Barnett-Vanes,2 Matthew J Carty,3 Peter McCulloch,4 Mahiben Maruthappu2

To cite: Trehan A, BarnettVanes A, Carty MJ, et al. The impact of feedback of intraoperative technical performance in surgery: a systematic review. BMJ Open 2015;5:e006759. doi:10.1136/bmjopen-2014006759 ▸ Prepublication history and additional material is available. To view please visit the journal (http://dx.doi.org/ 10.1136/bmjopen-2014006759). Received 17 October 2014 Revised 28 April 2015 Accepted 7 May 2015

ABSTRACT Objectives: Increasing patient demands, costs and
emphasis on safety, coupled with reductions in the length of time surgical trainees spend in the operating theatre, necessitate means to improve the efficiency of surgical training. In this respect, feedback based on intraoperative surgical performance may be beneficial. Our aim was to systematically review the impact of intraoperative feedback based on surgical performance. Setting: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews were searched. Two reviewers independently reviewed citations using predetermined inclusion and exclusion criteria. 32 data-points per study were extracted. Participants: The search strategy yielded 1531 citations. Three studies were eligible, which comprised a total of 280 procedures by 62 surgeons. Results: Overall, feedback based on intraoperative surgical performance was found to be a powerful method for improving performance. In cholecystectomy, feedback led to a reduction in procedure time ( p=0.022) and an improvement in economy of movement ( p<0.001). In simulated laparoscopic colectomy, feedback led to improvements in instrument path length ( p=0.001) and instrument smoothness ( p=0.045). Feedback also reduced error scores in cholecystectomy ( p=0.003), simulated laparoscopic colectomy ( p<0.001) and simulated renal artery angioplasty ( p=0.004). In addition, feedback improved balloon placement accuracy ( p=0.041), and resulted in a smoother learning curve and earlier plateau in performance in simulated renal artery angioplasty. Conclusions: Intraoperative feedback appears to be associated with an improvement in performance, however, there is a paucity of research in this area. Further work is needed in order to establish the longterm benefits of feedback and the optimum means and circumstances of feedback delivery.

Strengths and limitations of this study
▪ Systematic review, minimising likelihood of relevant papers being missed. ▪ Detailed extraction of data from studies. ▪ Very few relevant studies in the literature despite the importance of the topic.

1

Medical Sciences Division, University of Oxford, Oxford, UK 2 Imperial College London, London, UK 3 Harvard Medical School, Boston, USA 4 Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK Correspondence to Abhishek Trehan; abhishek.trehan@lincoln.ox. ac.uk

INTRODUCTION In light of increasing patient demands, costs and emphasis on safety, surgeons and their outcomes have become the subject of increased expectations and scrutiny.1 Coupled

with this, time spent in the operating theatre by surgical trainees is declining worldwide due to regulations that have reduced the legal number of working hours;2 3 this is particularly alarming in light of the now well-established relationship between surgical volume and surgical outcomes,4–6 and recent work has also directly linked intraoperative technical skill to complication and mortality rates.7 Such challenges necessitate increased efficiency of surgical training programmes such that an equivalent or superior level of surgical proficiency can be achieved in spite of the shorter length of time spent in the operating theatre. One means by which surgical skills acquisition could be enhanced is via the dissemination of feedback on intraoperative performance. Studies in medical students performing basic surgical skills such as suturing, knot tying and basic laparoscopic tasks, have demonstrated that feedback can improve skill acquisition.8–10 Additionally, proficiency in simulated laparoscopic salpingectomy is accelerated in medical students when they receive instructor feedback,11 and feedback improves colonoscopy performance in gastroenterologists.12 Thus, provision of feedback on intraoperative surgical performance to surgical trainees may also be associated with improved performance and/or a more rapid acquisition of skills, and hence formalised feedback should potentially serve as a key component of future surgical training programmes. Although feedback of intraoperative skill and technique can be a common occurrence in the operating theatre, the impact of this on performance and requirements for 1

Trehan A, et al. BMJ Open 2015;5:e006759. doi:10.1136/bmjopen-2014-006759

Open Access optimal training have thus far not been reviewed. We therefore conducted a systematic review to evaluate the impact of feedback of technical skill in the operating theatre and in the context of simulation. For the purpose of this study, feedback was defined as the provision of information pertaining to the operator’s surgical performance with the aim of improving subsequent performance. METHODS Data sources and search strategy The systematic review was conducted in accordance with PRISMA guidelines. A comprehensive search was undertaken to determine the impact of feedback on surgical performance via the Ovid SP interface. The following databases were searched from inception to February 2013: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews. We used two different domains of MeSH-terms and key words combined by ‘AND’, and within each domain the terms were combined by ‘OR’. The first domain contained terms related to surgical skill and performance, while the second contained terms related to the impact of feedback. A detailed search strategy can be found in online supplementary appendix 1. The search was limited to English publications with no other restrictions. Study selection Two reviewers independently reviewed citations and selected eligible studies based on predetermined inclusion and exclusion criteria. Publications were selected for review if they satisfied the following inclusion criteria: the article was published in a peer-reviewed journal; the article described a study involving surgical patients or simulation; the article investigated the impact of feedback of intraoperative surgical performance; the article used a statistical unit that was patient-focused or procedurefocused. The following exclusion criteria were applied to search results: the article was a conference abstract, editorial, letter, opinion, audit or review; the population studied was non-surgical (eg, pathology, medicine); the article described methods of feedback, not the impact of feedback; the article utilised a medical student population. Two authors (MM, AT) independently examined all retrieved articles for inclusion. Any disagreements over inclusion or exclusion were resolved by discussion between authors. References in relevant papers were also reviewed in order to identify any additional studies that may have been missed by the search strategy. Data extraction Thirty two data-points per study were extracted using a predesigned data collection form including: first author, year of publication, study aim, study type, study design (eg, prospective, retrospective, experimental, observational, crosssectional, longitudinal), study population, population setting (eg, hospital), surgical specialty, surgical procedure analysed, number of surgeons, types of feedback 2 dissemination, content of feedback, frequency of feedback, measured outcomes and interventions following feedback. The full data extraction from the studies can be found in online supplementary appendix 2. RESULTS Study identification and selection Our search yielded 1531 citations, of which 1185 articles were excluded. After detailed evaluation of the 346 remaining articles, three studies remained eligible, which comprised of a total of 280 procedures by 62 surgeons.13–15 A flow diagram of the search results is illustrated in figure 1. Study characteristics All three studies were performed on surgical trainees, one involving live cholecystectomy cases, one involving simulated laparoscopic colectomy and one involving simulated renal artery angioplasty. Two studies were two-armed RCTs (with one arm receiving feedback and the control arm receiving no feedback),13 15 while one study was a three-armed RCT (with one arm receiving expert feedback, another arm receiving non-expert feedback and one arm receiving no feedback).14 The studies included in this review are shown in table 1, and their basic characteristics are summarised in table 2. Feedback dissemination In all studies, feedback was delivered orally after each procedure.13–15 No written feedback was provided in any of the studies and one study required participants to selfassess their performance after each case in addition to receiving oral feedback.13 One study utilised video footage in facilitating feedback.15 Feedback contents The two examined studies involving simulation provided participants with feedback relevant to the exercise, including standard instrument metrics, procedural time and errors, accompanied with a description of correct methods where necessary.13 14 For the study of live surgery, feedback was facilitated by review of a videotape recording of the operation and a 60 min structured feedback session, during which technical deficiencies and possible errors were covered and instructions for improvement offered.15 One study provided benchmarking relative to peers13 and no studies provided surgeons with comparable data from the literature. In one study, feedback was provided solely by a single surgeon with significant operative and teaching experience in laparoscopy.15 One study assessed the impact of expert (consultant surgeons) and non-expert (inexperienced surgical trainees) instructor feedback.14 There was no difference to be found between expert and nonexpert feedback in all outcomes assessed other than error scores, which were lower when using the Vascular Interventional Surgical Trainer (VIST) error metrics and
Trehan A, et al. BMJ Open 2015;5:e006759. doi:10.1136/bmjopen-2014-006759

Open Access
Figure 1 Summary of search strategy for identification of relevant studies.

scoring during the operation ( p=0.009), but not so when using a custom, more extensive scoring sheet completed by a single expert on reviewing video footage of the procedures.14 Impact of feedback All three studies identified improvements in one or more of the outcomes assessed. Table 3 shows outcomes assessed across the three studies with associated p values.

In addition, the study of simulated renal artery angioplasty assessed procedure-specific outcomes including contrast volume (mL), fluoroscopic time (seconds), balloon placement accuracy (mm), residual stenosis and lesion coverage (%).14 Of these, balloon placement accuracy was shown to be significantly improved in those receiving feedback ( p=0.041).14 Although not reaching statistical significance, contrast volume utilised was 24.9 mL in control group, and 9.55 mL in those receiving feedback.14 While not demonstrated via statistical

Table 1 Overview of studies included and impact of feedback on performance Number of participating surgeons 28 18 16 Total number of cases 5 per surgeon 6 per surgeon 2 per surgeon Study design RCT RCT (three arms) RCT

Reference Boyle et al13 Boyle et al14 Grantcharov et al15

Country Ireland

Specialty

Surgical procedure

General surgery Simulated laparoscopic colectomy Ireland General surgery Simulated renal artery angioplasty Denmark General surgery Cholecystectomy

RCT, randomised controlled trial.

Trehan A, et al. BMJ Open 2015;5:e006759. doi:10.1136/bmjopen-2014-006759

3

Open Access
Table 2 Basic characteristics of studies included Study characteristics All studies Feedback dissemination Oral Written Self-assessment Feedback contents Outcomes Benchmarking relative to peers Comparable literaturereported figures Feedback frequency After each procedure Video footage utilisation Assessment participant performance Dissemination of feedback Number of studies References 3 3 0 1 3 1 0
13–15

13–15

13

13–15 13

3 2 1

13–15

14 15

15

methods, a smoother learning curve and earlier plateau in performance was noted in the group with feedback.14 DISCUSSION Our review included three studies assessing the impact of feedback of intraoperative surgical performance. Feedback was consistently found to be a powerful method for improving surgical performance in terms of operative metrics such as error scores and instrument movement metrics, as well as metrics specific to the procedure being undertaken. Feedback could thus represent a simple but powerful means by which efficiency and safety could be improved, thereby allowing for the attainment of surgical skills to a greater level of proficiency and/or in a shorter length of time in the context of training. This is of particular relevance as, with the exception of video and virtual reality simulator training, training methods known to enhance performance in the operating theatre are few and far between.16 17 Only three studies were included in this review, reflecting the dearth of research in this area despite the significant benefits that feedback could bring; there appear to be many studies in the literature that describe how to assess or rate technical skill,18–20 but very few that actually

assess how this data should be used. Limitations of this study include the fact that the search was conducted in February 2013, that conference abstracts were excluded and that study quality was not formally assessed. All three studies were randomised controlled trials, however, two of these three involved simulated, as opposed to live, procedures.13 14 Future studies should look to further assess the impact of feedback related to live surgery such that the broader implications of feedback can be appreciated. The impact of feedback on long-term skill acquisition was not studied; all studies only assessed surgical performance with between one and five procedures after the first feedback was provided. Thus, studies taking place over a longer time scale are necessary. It is also important to establish the clinical significance of feedback; none of the studies included assessed whether the improvement in technical skill was associated with an improvement in clinical outcomes, although one might suspect it would, particularly in light of recent findings that technical skills rated by experts based on video footage correlate with surgical outcomes.7 In one study, feedback included a review of a videotape recording and a 60 min structured feedback session with a senior surgeon.15 Although extensive feedback sessions have been suggested (but not shown) to be effective,21 provision of feedback in this manner may be resource intensive and hence cost- and time-effectiveness must also be considered. The finding from a study involving simulation that non-expert delivered feedback is still effective14 may broaden options for educationalists and time-pressed senior surgeons, although one must be careful not to implement counter-productive feedback initiatives. Although there are a few studies on this subject, all studies included in our analysis were randomised controlled trials. Given the consistent benefit of feedback demonstrated, this supports further research on this topic and implementation of structured intraoperative feedback initiatives. The small number of studies included in this review highlights the need for more substantive research in this area in order to establish the optimum means and circumstances of feedback dissemination such that standardised methods for future widespread implementation can be attained, and future studies should consider the effect of the following study variables

Table 3 Key outcomes in included studies Study Boyle et al13 Boyle et al14 Grantcharov et al15 Procedure Simulated laparoscopic colectomy Simulated renal artery angioplasty Cholecystectomy Procedure time – Ns 0.022 Instrument path length 0.001 – – Instrument Economy of smoothness movement 0.045 – – – – <0.001 Error scores <0.001 0.004 0.003

‘–’=outcome not assessed. p Values shown are p values for improvement in that outcome in feedback group when compared to control group with no feedback. Ns, not significant.

4

Trehan A, et al. BMJ Open 2015;5:e006759. doi:10.1136/bmjopen-2014-006759

Open Access 1. Source (oral/written), facilitator (expert/non-expert), frequency (every procedure/once daily/weekly/ monthly) and duration of feedback (months/years). 2. Surgeon involvement in feedback (either active or passive), standardised means of assessing surgical performance (which may be both generic and procedure-specific), content of feedback, timing of feedback relative to the procedure (intraoperatively/ postoperatively), and the opportunities available for discussion, correction and learning. 3. Benchmarking (relative to peers as well as to literature data) and feedback based on intraoperative recordings reviewed at a later time point. 4. Other interventions utilised, such as guidelines, education and review of instructional videos. The contributions of these interventions, and the additive effect they may have with feedback on performance and outcomes are poorly understood. It should also be borne in mind that in some circumstances or when delivered inappropriately, feedback may not be effective; for instance, although a number of studies in medical students have found feedback to improve acquisition of basic surgical skills,8–10 some have failed to find this,22 and the effect of feedback may plateau.23 24 Frameworks have been suggested in order to ensure appropriate dissemination of feedback,25 which is particularly important given the fact that trainees often feel they are provided with inadequate feedback despite senior surgeons feeling their feedback provision is adequate.26 27 In an era of increasing demands and scrutiny of surgeons in which surgical trainees are simultaneously spending less time in the operating theatre, methods to improve the efficiency of surgical performance are needed. The findings from this review suggest that feedback of intraoperative performance is an effective means by which this might be achieved; however, despite the potential impact, there is a paucity of research in this area, and further work is needed in order to establish the optimum circumstances and means by which feedback can be delivered in a timeeffective and cost-effective manner.
Contributors AT, MM, AB-V were involved in the systematic search, manuscript composition, proofing. MJC, PM were involved in the conception, design, drafting, proofing. Competing interests None declared. Provenance and peer review Not commissioned; externally peer reviewed. Data sharing statement The search criteria and data extracted from the studies are available in the Appendices. Open Access This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work noncommercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http:// creativecommons.org/licenses/by-nc/4.0/
2. Breen KJ, Hogan AM, Mealy K. The detrimental impact of the implementation of the European working time directive (EWTD) on surgical senior house officer (SHO) operative experience. Ir J Med Sci 2013;182:383–7. Kairys JC, McGuire K, Crawford AG, et al. Cumulative operative experience is decreasing during general surgery residency: a worrisome trend for surgical trainees? J Am Coll Surg 2008;206:804–11; discussion 811–3. Sosa JA, Bowman HM, Tielsch JM, et al. The importance of surgeon experience for clinical and economic outcomes from thyroidectomy. Ann Surg 1998;228:320–30. Carty MJ, Chan R, Huckman R, et al. A detailed analysis of the reduction mammaplasty learning curve: a statistical process model for approaching surgical performance improvement. Plast Reconstr Surg 2009;124:706–14. Birkmeyer JD, Stukel TA, Siewers AE, et al. Surgeon volume and operative mortality in the United States. N Engl J Med 2003;349:2117–27. Birkmeyer JD, Finks JF, O’Reilly A, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med 2013;369:1434–42. Rogers DA, Regehr G, Howdieshell TR, et al. The impact of external feedback on computer-assisted learning for surgical technical skill training. Am J Surg 2000;179:341–3. Porte MC, Xeroulis G, Reznick RK, et al. Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. Am J Surg 2007;193:105–10. Van Sickle KR, Gallagher AG, Smith CD. The effect of escalating feedback on the acquisition of psychomotor skills for laparoscopy. Surg Endosc 2007;21:220–4. Strandbygaard J, Bjerrum F, Maagaard M, et al. Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized trial. Ann Surg 2013;257:839–44. Harewood GC, Murray F, Winder S, et al. Evaluation of formal feedback on endoscopic competence among trainees: the EFFECT trial. Ir J Med Sci 2008;177:253–6. Boyle E, Al-Akash M, Gallagher AG, et al. Optimising surgical training: use of feedback to reduce errors during a simulated surgical procedure. Postgrad Med J 2011;87:524–8. Boyle E, O’Keeffe DA, Naughton PA, et al. The importance of expert feedback during endovascular simulator training. J Vasc Surg 2011;54:240–248 e1. Grantcharov TP, Schulze S, Kristiansen VB. The impact of objective assessment and constructive feedback on improvement of laparoscopic performance in the operating room. Surg Endosc 2007;21:2240–3. Gurusamy K, Aggarwal R, Palanivelu L, et al. Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg 2008;95:1088–97. Dawe SR, Windsor JA, Broeders JA, et al. A systematic review of surgical skills transfer after simulation-based training: laparoscopic cholecystectomy and endoscopy. Ann Surg 2014;259:236–48. Gofton WT, Dudek NL, Wood TJ, et al. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med 2012;87:1401–7. van Hove PD, Tuijthof GJ, Verdaasdonk EG, et al. Objective assessment of technical surgical skills. Br J Surg 2010;97:972–87. Vassiliou MC, Feldman LS, Fraser SA, et al. Evaluating intraoperative laparoscopic skill: direct observation versus blinded videotaped performances. Surg Innov 2007;14:211–16. Middaugh DJ. The 2-minute warning part v: conduct a post-game analysis and press conference. Medsurg Nurs 2012;21:177–8. Snyder CW, Vandromme MJ, Tyra SL, et al. Effects of virtual reality simulator training method and observational learning on surgical performance. World J Surg 2011;35:245–52. O’Connor A, Schwaitzberg SD, Cao CG. How much feedback is necessary for learning to suture? Surg Endosc 2008;22:1614–19. Stefanidis D, Korndorffer JR Jr, Heniford BT, et al. Limited feedback and video tutorials optimize learning and resource utilization during laparoscopic simulator training. Surgery 2007;142:202–6. Ahmed M, Arora S, Russ S, et al. Operation debrief: a SHARP improvement in performance feedback in the operating room. Ann Surg 2013;258:958–63. Sender Liberman A, Liberman M, Steinert Y, et al. Surgery residents and attending surgeons have different perceptions of feedback. Med Teach 2005;27:470–2. Jensen AR, Wright AS, Kim S, et al. Educational feedback in the operating room: a gap between resident and faculty perceptions. Am J Surg 2012;204:248–55.

3.

4. 5.

6. 7. 8. 9.

10. 11.

12. 13. 14. 15.

16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26.

REFERENCES
1. Tavare A. Where are we with transparency over performance of doctors and institutions? BMJ 2012;345:e4464.

27.

Trehan A, et al. BMJ Open 2015;5:e006759. doi:10.1136/bmjopen-2014-006759

5