Implementation of an electronic surgical referral service. Collaboration, consensus and cost of the surgeon – general practitioner Delphi approach
Authors Augestad K, Revhaug A, Johnsen R, Skrøvseth S, Lindsetmo R
Received 23 April 2014
Accepted for publication 28 May 2014
Published 9 September 2014 Volume 2014:7 Pages 371—380
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Knut Magne Augestad,1–3 Arthur Revhaug,1,3 Roar Johnsen,4 Stein-Olav Skrøvseth,2 Rolv-Ole Lindsetmo1,3
1Department of Gastrointestinal Surgery, 2Department of Integrated Care and Telemedicine, University Hospital North Norway, Tromsø, Norway; 3Department of Colorectal Surgery, University Hospitals Case Medical Center, Cleveland, Ohio, USA; 4Department of Public Health and General Practice, Norwegian University of Science and Technology, Trondheim, Norway
Background: Poor coordination between levels of care plays a central role in determining the quality and cost of health care. To improve patient coordination, systematic structures, guidelines, and processes for creating, transferring, and recognizing information are needed to facilitate referral routines.
Methods: Prospective observational survey of implementation of electronic medical record (EMR)-supported guidelines for surgical treatment.
Results: One university clinic, two local hospitals, 31 municipalities, and three EMR vendors participated in the implementation project. Surgical referral guidelines were developed using the Delphi method; 22 surgeons and seven general practitioners (GPs) needed 109 hours to reach consensus. Based on consensus guidelines, an electronic referral service supported by a clinical decision support system, fully integrated into the GPs' EMR, was developed. Fifty-five information technology personnel and 563 hours were needed (total cost 67,000 £) to implement a guideline supported system in the EMR for 139 GPs. Economical analyses from a hospital and societal perspective, showed that 504 (range 401–670) and 37 (range 29–49) referred patients, respectively, were needed to provide a cost-effective service.
Conclusion: A considerable amount of resources were needed to reach consensus on the surgical referral guidelines. A structured approach by the Delphi method and close collaboration between IT personnel, surgeons and primary care physicians were needed to reach consensus.
Keywords: hospital referrals, surgery, patient pathways, process health care assessment, electronic medical record
Busy hospital clinics and increased waiting time for surgical treatment have made it necessary to develop new referral routines and to improve the patient flow and coordination between primary and secondary care. Poor coordination between different levels of care plays a central role in determining health care quality and cost.1–3 To improve coordination, systematic structures, guidelines (clinical decision support [CDS] systems), and processes for creating, transferring, and recognizing information are needed to facilitate primary–secondary care communication and referral routines.1 Improved referral routines have the potential to decrease waiting time and to decrease the workload for hospital clinics.4–7 CDS systems can be used as a tool to improve referral quality and the coordination of care. CDS systems provide clinicians with patient-specific assessment tools or guidelines to aid their clinical decision-making and have been shown to improve health care quality.8,9 CDS systems improve prescribing practices, reduce serious medication errors, enhance the delivery of preventive care services, improve adherence to guidelines, and result in lasting improvements in clinical practice.10–13
In this research project we focused on referrals to outpatient surgery. Outpatient surgery is a cost-effective solution for certain surgical patients because it decreases the time between the referral and the surgery and subsequently improves the patient flow.5,14–19 Furthermore, outpatient surgery decreases the burden of internal hospital logistics by decreasing the number of administrative personnel involved in the referral process.14–19
In 2007, we initiated an outpatient surgical referral program, aiming to decrease waiting times and improve the patient flow at the outpatient clinics and operational theatres. We wanted to implement a CDS-supported referral system in the general practitioners’ (GPs’) electronic medical record (EMR) and thereby improve surgical referral quality, improve the coordination of care, and increase the rate of outpatient surgery (one stop surgery). The objective of this observational trial was to describe the Delphi method and resources needed (money, time, and personnel) to develop and implement a CDS-supported surgical referral system in the GPs’ EMR.
This prospective observational study took place between June 2007 and June 2011 and was divided into three main phases: 1) reaching guideline consensus on the content of a CDS-supported referral service; 2) developing the CDS-supported referral software; and 3) large-scale implementation of an EMR-integrated referral service. The use of resources (surgeons, GPs, administrative staff, and technology) was prospectively registered. The primary objective of the survey was to describe the primary care physician–surgeon consensus process by the Delphi method. The secondary objective was to perform an estimate of the number of referrals needed to establish a cost-effective referral service (ie, cost of consensus and implementation = potential cost savings by improved referral routines).
The Department of Digestive Surgery is divided into clinics located at three different hospitals. All of these clinics perform the surgical procedures included in the trial (pilonidal sinus: Bascom plasty; inguinal hernia: Lichtenstein repair; gallstone disease: laparoscopic cholecystectomy). In the county of Troms, there are 31 municipal GP practices with 139 GPs. All of the GP practices are connected to the Norwegian National Health Network, which sends referrals electronically to hospitals.
Patients were included in the analyses if they were diagnosed and referred to hospital by their GP for an inguinal hernia, umbilical hernia, sinus pilonidalis, or gallstone disease requiring surgical treatment.
Consensus of a guideline-supported referral system
A modified Delphi approach was used,21 divided into the following phases: generation of referral recommendations, a panel meeting of senior surgeons, a first consensus round with all surgeons (review results), a second consensus round with all surgeons (ratings), and the evaluation of the consensus. Relevant evidence regarding the different conditions was presented, and all of the participants in the process were asked to include only the “need to know” information about the surgical procedure. This information includes:
- Disease-specific guidelines: These guidelines included signs and symptoms, surgical indications and contraindications, blood samples, and radiological examinations needed prior to surgery (laparoscopic cholecystectomy).
- Standardized referral forms: The referral forms included information needed for anesthesia (mandatory), information needed for surgeons (mandatory), and additional information.
- Patient information: This information included the time of surgery, the surgical procedure, the instructions for the patient to follow after surgery, and the possible complications of surgery.
After we reached internal hospital department consensus, the referral templates and guidelines were discussed with a panel of three experienced GPs. The GPs provided comments and suggestions for further improvement of the referral forms, disease-specific guidelines, and patient information. All changes suggested by the GPs were then presented and accepted by the head of the surgical department.
All of the necessary resources (hours) used to reach consensus were prospectively registered.
Development of the CDS-supported EMR referral software
Two EMR companies were involved in the development, piloting, and implementation of the CDS solution. A third company was responsible for the software solution making communication between EMR 1 and EMR 2 possible. The development and implementation of the technology were adjusted to incorporate the features of CDS systems that have been proven effective for changing clinical practice.20 All of the necessary resources were prospectively registered.
Referral piloting and large-scale implementation
A primary care office was chosen as the pilot office. The CDS-supported referral software was discussed with the GPs, and improvements were made after receiving feedback from the GPs. Test referrals were sent from the GP office to the hospital until all of the technological problems had been resolved. Information technology (IT) experts integrated the referral software into the GPs’ EMR. The software was fully integrated into the GPs’ EMR and triggered by the corresponding international classification of primary care (ICPC) codes.
Comparison of referral cost-effectiveness
Two referral routes were compared: a) the traditional referral route where all patients are referred by the GP to the outpatient clinic and the outpatient clinic surgeon then refers the patients to day case surgery; and b) The GP refers directly to day case surgery (Figure 1). The sensitivity analysis used both a societal and a hospital perspective. The referral costs included administrative hospital costs, the examination in the surgical outpatient department (surgeon time, nurse time), the surgical treatment (surgeon time, anesthetist time, and nurse time), the unused surgical outpatient capacity caused by incorrect referrals (both for new and traditional service), and the patients’ travel costs. All of the cost estimates (Table S1) were created separately and compared. We assumed that 10% of the one-stop patients would be denied surgery because of inconsistent referral information and that 20% of the referred patients were on sick leave at the time of referral. The number of patients needed to break even (the threshold, ie, the point at which the cost savings from a one stop surgery (OSS) referral service equal the cost of implementation) was estimated. This value was obtained using the following equation:
We defined a potential decrease in waiting time as “total waiting time – intrahospital waiting time.” Prehospital waiting time was defined as the time between the referral and the first consultation at the surgical outpatient clinic. Data from the national waiting list register was extracted (http://www.frittsykehusvalg.no/start/) to perform this analysis. Intrahospital waiting time was defined as the time between the first consultation at the outpatient clinic and the surgical treatment. The NOMESCO (Nordic Medico-Statistical Committee) Classification of Surgical Procedures codes were extracted from the hospital’s EMR database to perform this analysis. To estimate the potential frequency of patients referred by the new service, consecutive referrals (an 18-month time interval) for sinus pilonidalis (International Classification of Diseases [ICD] 10: L05.0 and L05.9), gallstone disease (ICD 10: K80.2), inguinal hernia (ICD 10: K40.0, K40.2, K40.9), and umbilical hernia (ICD 10: K42.0, K42.9) were reviewed by two authors (Knut Magne Augestad [KMA] and Rolv-Ole Lindsetmo [ROL]). Cost data were converted from Norwegian kroner (NOK) into British pounds at the rate of 1 £ =9.39 NOK (based on the exchange rage according to the Norwegian National Bank on June 27, 2012). The BMJ guidelines for health-related economic evaluations were followed.22
The Delphi process
All of the surgeons approved the guidelines, and consensus was reached after a 3-month period. Overall, 109 hours were needed to reach guideline consensus, to an estimated cost of 10,800 £ (Tables 1 and 2).
Table 1 The Delphi approach of referral consensus for surgeons and GPs
Development and implementation of the referral software
IT personnel (n=24) from three different commercial vendors participated in the development of the CDS software, and the overall cost of the software development was estimated at 32,652 £. A total of 140 hours were needed to pilot and implement the software for 139 GPs, at a total cost of 23,581 £. The total cost of work phases 1 to 3 was 67,075 £ (Table 3).
Main effects of new referral routines
We reviewed 700 referrals (ie, 18 months consecutive referrals); 189 (27%) were classified as well-suited for the referral service. The median waiting time to the examination in the outpatient clinic was 84 days, and the median time from the examination until the outpatient surgery was 101 days (ie, the overall median waiting time was 185 days). The estimated cost savings from the referral service were 133 £ per patient from the hospital perspective and 18,909 £ from the societal perspective (Table 4).
A tornado chart was employed to analyze the uncertainties of referral implementation cost. Small variations in the cost of CDS technology development (phase 2) will have large effects on the overall cost compared with the other cost elements we analyzed (Figure 2).
Summary of findings
To our knowledge, this survey is the first to address the time, resources, and funding needed to develop and implement an electronic hospital referral service supported by a CDS system. The decision support tool was fully integrated into the GPs’ EMR and was triggered by the referring ICPC code. A total of 84 persons (surgeons, GPs, administrative personnel, and health IT personnel) and 550 hours (from the beginning of the project) were needed to implement the CDS referral software in the EMR for 139 GPs. The overall cost of the surgical Delphi consensus process, the development of the CDS technology, and the implementation of the technology was 67,000 £. A sensitivity analysis showed that unexpected technology obstacles had the greatest impact on the overall cost of CDS implementation. Finally, we estimated the effects of an integrated CDS referral system for patients and the health care system. This surgical referral system has the potential to reduce the waiting time for surgery by 101 days. Of the referrals for selected surgical conditions, 27% were well-suited for a one-stop surgical service, which means that the service was cost-effective (from the hospital perspective) after 504 CDS-supported referrals. Because of the decreased waiting time for surgical treatment, the length of sick leave will decrease, resulting in societal savings of approximately 1,809 £ for each patient who is referred to an expedited one-stop surgical service.
Comparison with the existing literature
Effective communication is essential for coordinating health care; however, patient care is often compromised by poor communication between primary and secondary care services.23–25 Poor coordination between primary and secondary care services contributes to avoidable patient morbidity and mortality.26 Thus, effective communication between primary care physicians and hospital specialists regarding patient referrals, consultations, and treatment is necessary to improve patient outcomes and physician satisfaction.27 Poor communication plays a central role in determining health care quality and cost, and systematic structures and tools (like CDS systems) are needed to facilitate primary–secondary care communication.1–3
To address the deficiencies in communication and the coordination of care, health care is increasingly turning to CDS systems, which provide clinicians with patient-specific assessments or recommendations to aid in their clinical decision-making and process management.20 CDS systems have been shown to reduce errors and improve the coordination of care.20 Several features of CDS systems are associated with improvements in clinical practice (automatic provision, integration with the EMR, and provision of recommendations). Of these features, automatic provision of decision support as part of the clinician’s workflow is the single most important feature, with a reported odds ratio of 105.20,28 In this project, we focused on similar CDS features associated with improvements in clinical practice. When we designed the user interface of the CDS, the CDS was triggered by the referral ICPC code, thus becoming an automatic part of the GPs’ referral workflow.
The rate of referrals has been growing rapidly, with potential implications for health care spending.29 There are no widely accepted guidelines for referring patients. Despite the push toward evidence-based decision-making, the referral of patients is driven primarily by the practice patterns of physicians, not by the patient case mix. Patient referrals can profoundly change the nature and cost of the care that patients receive. We need to understand the benefits and risks of specific referral patterns, as we do with any intervention.30,31
Strengths and limitations
This trial has some strengths. First, many trials have evaluated the clinical effects of CDS systems; however, few of these trials have assessed the effects of the systems on the coordination of care between different service levels of health care. Second, most studies that have assessed referral systems have been conducted without a fully integrated EMR system implemented in primary and secondary care. Although two separate EMR vendors support our EMR systems, they are fully integrated in the entire health care system, with referrals and text messages sent electronically between the two systems. This integration increases the potential of a CDS-supported referral service because it offers greater flexibility to adjust the CDS system to new routines or new referral patterns. The CDS referral system can be updated from one central server located at the university hospital (ie, no technical personnel have to travel to the GPs’ offices).
There are also limitations to this study. First, the long-term implementation and routine use of CDS systems remains challenging. In a recently published review of randomized trials on decision support systems, only 18% discussed long-term implementation and long-term use.32 At the present time, we have not succeeded in fully incorporating the referral service into routine use, a problem that has also been reported in other trials.33,34 We acknowledge that there is a need to improve the GP reimbursement policy because of the increase in the referral workload. Second, as a quality improvement project, there are certain limitations that are inherent to our work.
The economic analysis has several limitations. There are uncertainties attached, especially with regard to the proportion of patients that can return faster back to work caused by decreased waiting times and the proportion of patients who are rejected because of improper clinical information in the referral. These uncertainties must be assessed using future prospective surveys.
Implications for patients and decision-makers
Higher-quality surgical referrals will have great implications for patients. We have analyzed the implications of a standardized surgical referral service, which produces a shorter waiting time before surgical treatment and subsequently a decrease in sick leave. For other conditions, high-quality referrals and closed communication loops improve the coordination of care. This means that there will be fewer hospital admissions, fewer radiologic exams, fewer blood samples, and more informed health care providers offering better continuity of care.1,3,24,25 These aspects are especially important for patients with multimorbidity and chronic diseases, for whom new strategies are needed to coordinate care.35 For hospital decision-makers, high-quality referrals and better communication between GPs and hospital specialists might have large implications for the quality of care. Our hospital trust receives approximately 140,000 referrals annually, imposing a huge burden in terms of coordination and the workload for physicians and administrative personnel. A CDS referral system can improve coordination, decrease hospital travel time by improving process management, and reduce the burden on busy outpatient clinics.27 Improved coordination has been shown to have large economic implications.2 However, better technological coordination of EMR systems and greater standardization of patient pathways are needed. Our analyses show that the waiting time for surgery will decrease, thus decreasing the burden for patients and resulting in cost savings for society as the duration of sick leave decreases. Finally, fewer resources will be used by admitting hospitals because of the improvements in logistics and the coordination of care.
As referral rates and reports of poor coordination between primary and secondary care increase, quality improvement projects focusing on referral quality are needed.2,29,31 CDS systems have the potential to improve communication and coordination between primary and secondary care and thus increase the OSS rate.32 This study has two main conclusions. First, many resources (personnel, time, and funding) were needed to implement a CDS-supported referral service, mainly because of the time-consuming process of establishing a medical consensus, unexpected technology obstacles, and organizational obstacles in the interface between primary and secondary care. Increased efforts are needed to make CDS referral systems more attractive for GPs, and incentives must respond to the increase in the referral workload.
This work was funded by the North Norwegian Health Authorities Research Fund (grant number 40614).
We thank Heidi Jacobsen at the Norwegian National Centre of Integrated Care and Telemedicine for her administrative support during the project.
The internal review board of University Hospital North Norway provided ethical approval for this project.
All authors contributed toward data analysis, drafting, and revising the paper and agree to be accountable for all aspects of the work.
The authors report no conflicts of interest in this work.
Foy R, Hempel S, Rubenstein L, et al. Meta-analysis: effect of interactive communication between collaborating primary care physicians and specialists. Ann Intern Med. 2010;152(4):247–258.
Bodenheimer T. High and rising health care costs. Part 3: the role of health care providers. Ann Intern Med. 2005;142(12 Pt 1):996–1002.
Chen A, Yee H. Improving primary care-specialty care communication: lessons from San Francisco’s safety net: comment on “Referral and consultation communication between primary care and specialist physicians”. Arch Intern Med. 2011;171(1):65–67.
Canadian Agency for Drugs and Technologies in Health. Surgical Referral and Appropriateness of Surgery: Clinical Evidence, Best Practices and Guidelines. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2011. Available from: http://www.cadth.ca/en/publication/2962. Accessed June 25, 2014.
Dennison J, Eisen S, Towers M, Ingham Clark C. An effective electronic surgical referral system. Ann R Coll Surg Engl. 2006;88(6):554–556.
Akbari A, Mayhew A, Al-Alawi MA, et al. Interventions to improve outpatient referrals from primary care to secondary care. Cochrane Database Syst Rev. 2008CD005471.
Clarke A, Blundell N, Forde I, et al. Can guidelines improve referral to elective surgical specialties for adults? A systematic review. Qual Saf Health Care. 2010;19(3):187–194.
Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280(15):1339–1346.
Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–1238.
Bennett JW, Glasziou PP. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Med J Aust. 2003;178(5):217–222.
Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999;6(2):104–114.
Lyman JA, Cohn WF, Bloomrosen M, Detmer DE. Clinical decision support: progress and opportunities. J Am Med Inform Assoc. 2010;17(5):487–492.
Emery JD. Effect of computerised evidence based guidelines. Computer support is complex intervention. BMJ. 2003;326(7385):394; author reply 394.
Patkar V, Hurt C, Steele R, et al. Evidence-based guidelines and decision support services: A discussion and evaluation in triple assessment of suspected breast cancer. Br J Cancer. 2006;95(11):1490–1496.
Trickett JP, Donaldson DR, Bearn PE, Scott HJ, Hassall AC. A study on the routes of referral for patients with colorectal cancer and its affect on the time to surgery and pathological stage. Colorectal Dis. 2004;6(6):428–431.
Julian S, Naftalin NJ, Clark M, et al. An integrated care pathway for menorrhagia across the primary-secondary interface: patients’ experience, clinical outcomes, and service utilisation. Qual Saf Health Care. 2007;16(2):110–115.
Eltahir A, Jibril J, Squair J, et al. The accuracy of ‘one-stop’ diagnosis for 1,110 patients presenting to a symptomatic breast clinic. J R Coll Surg Edinb. 1999;44(4):226–230.
Witcher TP, Williams MD, Howlett DC. “One-stop” clinics in the investigation and diagnosis of head and neck lumps. Br J Oral Maxillofac Surg. 2007;45(1):19–22.
McKessock L, Smith BH, Scott A, et al. A randomized controlled trial of direct access for laparoscopic sterilization. Fam Pract. 2001;18(1):1–8.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005; 330(7494):765.
Loblaw DA, Prestrud AA, Somerfield MR, et al; Americal Society of Clinical Oncology Clinical Practice Guidelines. American Society of Clinical Oncology Clinical Practice Guidelines: formal systematic review-based consensus methodology. J Clin Oncol. 2012;30(25):3136–3140.
Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ. 1996;313(7052):275–283.
Torjesen I. Care of IBD patients compromised by poor communication between primary and secondary care. BMJ. 2012;344:e2675.
Gandhi TK, Sittig DF, Franklin M, Sussman AJ, Fairchild DG, Bates DW. Communication breakdown in the outpatient referral process. J Gen Intern Med. 2000;15(9):626–631.
Shakib S, Philpott H, Clark R. What we have here is a failure to communicate! Improving communication between tertiary to primary care for chronic heart failure patients. Intern Med J. 2009;39(9):595–599.
McGlynn E, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–2645.
O’Malley AS, Reschovsky JD. Referral and consultation communication between primary care and specialist physicians: finding common ground. Arch Intern Med. 2011;171(1):56–65.
Tamblyn R, Huang A, Taylor L, et al. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15(4):430–438.
Barnett ML, Song Z, Landon BE. Trends in physician referrals in the United States, 1999–2009. Arch Intern Med. 2012;172(2):163–170.
Salas E, Wilson KA, Murphy CE, King H, Salisbury M. Communicating, coordinating, and cooperating when lives depend on it: tips for teamwork. Jt Comm J Qual Patient Saf. 2008;34(6):333–341.
Katz MH. How can we know so little about physician referrals? Arch Intern Med. 2012;172(2):100.
Augestad KM, Berntsen G, Lassen K, Bellika JG, Wootton R, Lindsetmo RO; Study Group of Research Quality in Medical Informatics and Decision Support (SQUID). Standards for reporting randomized controlled trials in medical informatics: a systematic review of CONSORT adherence in RCTs on clinical decision support. J Am Med Inform Assoc. 2012;19(1):13–21.
Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002; 325(7370):941.
Hetlevik I, Holmen J, Krüger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system. Int J Technol Assess Health Care. 2000;16(1):210–227.
Barnett K, Mercer SW, Norbury M, Watt G, Wyke S, Guthrie B. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet. 2012;380(9836):37–43.
Table S1 Details of the unit costs assigned to health care resource use data