Simulation training in neurosurgery: advances in education and practice

Introduction

Education in medicine in the past decade has progressively started to adopt virtual reality (VR) as a primary tool in training learners. A tool that can be exploited for both its cognitive task analysis and technical skills training, VR has been assuming a paramount role in medical and, more specifically, surgical training. Continued rapid development of medical technology, portable electronic devices and software with easy-to-use, friendly interfaces has seamlessly paved a path for implementation of VR for training across a wide array of applications.¹

With its unique combination of a high-yield/low-risk investment, VR embraces learners of all types. From practicing surgeons, residents and medical students to nurses and allied health professionals, VR can strengthen skills from basic knowledge and core thought processes to the most complex technical nuances in neurosurgical procedures. Moreover, it can also serve as a useful tool for educators to objectively review assessments, track learning and follow improvement of learners.

Most of the core foundations in medical education, particularly in the USA, can be traced back to Abraham Flexner, who authored the self-titled Flexner Report in 1910.² Credited in this report were advances of the “self-registering” thermometer, stethoscope, microscope and laboratory services of the times. These were noted to “extend the physician’s range”. The report eloquently noted the importance of teaching and research and the necessity of establishing a modus vivendi. With this article, like many others, we advocate the collaboration of neurosurgical education and VR. The politically controversial excerpts notwithstanding, the rudimentary principles and general educational values that resound in the Flexner Report are timelessly applicable. With profound literature unveiling striking estimates of nearly 100,000 hospital deaths per year in the USA due to medical errors,³ it behooves us to take advantage of our era and optimize medical and neurosurgical training. Aiming to reduce complications and medical errors in neurosurgery by developing VR training tools calls for not only precise, but also accurate representations of real-life situations and tactile feedback.

Cruces of the neurosurgical training paradigm

Neurosurgical training programs share many obstacles with other surgical training programs when it comes to resident education. Most importantly, educators must balance optimizing learner education with appropriately graduated learner autonomy against the goal of patient safety.

The recent changes by the Accreditation Council for Graduate Medical Education (ACGME) which instituted an 80-hour work-week for residents makes a strong call for a refurbished training paradigm. Historically, residents had time-intensive apprenticeship models, which were reasoned to gain necessary experience. However, some argue that less time allowed in the hospital equates to less time spent gaining the surgical training that one requires.⁴

With much debate, however, this also creates more time for an adequate work-life balance and encourages involvement in scholarly academic extracurricular projects.⁵ Though a new obstacle in neurosurgery residency training, it is one of the many challenges a neurosurgery training program faces.

More so, in the last decade, health care administrators, physicians and nonphysicians alike have been advocating for transparency of outcomes, complications, expenses and patient satisfaction.^6,7 A national mission statement for patient-centered care has arrived at the forefront of the development of institutional policy and safety measures. Physicians now perhaps more than ever operate with these factors in mind. These factors, though uniformly and holistically endorsed, pose challenges to neurosurgery training programs. To best meet patient needs, neurosurgeons need to perform increasingly complex and risky procedures without compromising patient care, all while training new learners to accomplish the same. The present time is better than any to utilize the technologies at hand. Neurosurgery needs to explore and develop VR methods for its trainees to garner the experience and comfort with its increasingly complex procedures that are advancing rapidly.

In neurosurgical training, residents and medical students can enjoy and absorb a vast array of knowledge from various learning opportunities. The operating room (OR) setting offers the most fruitful of experiences; however, the greatest yield for the learner must be from tactful and experienced educators. Orchestrating a successful, smooth surgery while building on a resident’s observational learning does not come easy. Residents in every program have access to large libraries with more than enough literature to satisfy curious minds. This also is monitored by the ACGME. Clinical rounds, especially in the neurosurgical intensive care unit, are filled with learning opportunities that also have been supported by off-site patient simulations with specific critical fast-paced scenarios. These patient simulations replicate the intensive care unit or the OR setting to help the resident learn to deal with critically ill patients or with intraoperative complications. Cadaver labs are available to many residency programs, but are not easily and readily accessible in many places. These resources help in taking the resident away from areas of high stress and use a more relaxed environment to build learning. The courses are key in advancing cognitive task rehearsals, but the tactile and technical skills may plateau with less-than-ideal simulations or with bloodless/poorly preserved cadavers.

Advances in imaging modalities such as magnetic resonance imaging (MRI) with diffusion tensor imaging with fiber tracking sequences, functional MRIs (fMRIs) and 3-D reconstructions allow the learner to conceptualize anatomy and pathologies like never before.^8,9 These types of software applications already in daily use by health care providers for surgical planning with goals of improving patient outcome can be essential tools for improving resident learning. Preoperative discussions between the educators and residents about imaging findings, surgical options and anatomic and functional relationships can explore key concepts to evaluate residents. These, among others, are examples of current VR integrations that highlight the importance of cognitive analytics in today’s neurosurgical training paradigm.

Background of simulation in medicine

Medical training has always relied on simulation in some capacity. To better understand human anatomy, early pioneers in the field first used cadavers. Careful dissection of these cadavers led curious minds to a better understanding of disease and pathology. In modern medical training, students are introduced to actors in simulated patient interactions. The goal of these interactions is to teach students how to interact with patients, how to take histories, how to perform a physical exam thoroughly while maintaining a patient’s modesty – all before they begin working in a clinical setting and meet actual patients.^10–17

It makes sense that practicing skills and encounters would help improve experiences and outcomes. This same thought was applied to the surgical field with simulators. General surgery has perhaps the most well-established and researched use of simulators. General surgeons, to become familiar with tools that offer a different sense of visual and tactile feedback, have used laparoscopic simulators. Studies have been performed looking at groups that can practice on a VR simulator before a laparoscopic cholecystectomy. Results show that the groups allowed to practice are less likely to have errors or make critical mistakes and complete the procedure quicker.¹⁸

A review of simulators used in general surgery performed by Carter et al determined that there are several criteria needed for a simulator to be effective.¹⁸ In order to judge a simulator and its ability to act as a true substitute for real-world scenarios, it must have reliability and validity. The goal for a simulator is to achieve several levels of validity: face, content, construct and predictive. Face validity is defined as the degree to which a procedure, especially a physiologic test or assessment, appears effective in terms of its stated goal. Content validity is the ability of the simulator to represent all facets of a given construct. Construct validity is defined as the ability to distinguish between subjects with different levels of experience. Predictive validity is measured by the correlation between the performance measured and the technical ability in vivo. To summarize, an ideal simulator must be able to perform the task it advertises, in its entirety, separate novices from experts and have these learned skills be transferrable to patients. This would be the perfect situation; however, we have yet to achieve this degree of simulation.

Simulators can be classified in various ways and a universal classification scheme remains elusive reflecting different aspects that may be emphasized independent of the point of view.^19,20 Numerous attempts at classification of simulators have been made in the past. They invariably are either overly broad and lack specificity or end up focusing on narrow aspects of simulation only. The ideal scheme would give the user adequate information about the technology used in the planning and the design of the simulator, and an indication of how the simulator may be utilized. Other desirable characteristics are that the classification system should be easy to remember. Such a classification scheme should map existing simulators in a rather straightforward way, but it would be even more desirable if it provides a roadmap both by demonstrating gaps and directions that research and development need to take.²⁰ Understanding the properties of simulators may aid the decisions leading to the choice of an appropriate device for teaching, training and for assessing a particular skill or task (Figure 1).

Figure 1 Classification of simulators. Note: Data from Pott et al.20

Simulation in training

Simulators have the power of automation. A cause and effect relationship with computer integration allows for immediate feedback and production of score reports. Several metrics such as scales of precision and accuracy are set by predetermined values that represent ideal dexterous technique.^21–25 Variations from these values produce feedback reports. The time to completion, number of errors performed and the need for assistance can be measured. In addition to focusing on technical skills and cognitive analysis, simulations allow learners to improve upon other qualities that mold able neurosurgeons. These include patient communication, teamwork, judgment and leadership.

These nontechnical areas address the six core competencies that the ACGME has laid out for resident evaluation: medical knowledge, patient care, practice-based learning and improvement, interpersonal and communication skills, professionalism and system-based practice. By engaging in simulations, residents can not only improve their skills in the OR, but also address core competencies that are needed to meet ACGME milestones.

Simulation in neurosurgery

There has been a significant interest in the field of neurosurgery to explore and utilize medical simulation and VR for purposes of training, skill maintenance, refinement, demonstration and assessment (Figures 2 and 3). Using PubMed keywords and phrases, which were also included into a MeSH search, yielded a total of 124 results and of these, we reference a selection of publications deemed relevant to this article.^{14,16,23,25–80}

Figure 2 Number of citations in PubMed covering the concepts of medical simulation and virtual reality in neurosurgery.

We utilized the PubAtlas⁸¹ interface for PubMed to derive the results of the pairwise intersections between search terms. This allows extending PubMed with a literature analysis interface. Correspondingly, the basic form of a literature map that PubAtlas provides explores associations among sets of terms in a graphical format. A high-level concurrent concept overview between the concepts of “neurosurgery”, “medical simulation” and “virtual reality” can be developed (Figure 3). We allowed the algorithm to explore the MeSH terms and find subassociations. To explore how these terms may correlate with the traditional subspecialties in neurosurgery, we added terms for these, for example, “trauma”, “vascular” and other similar terms. We subsequently allowed for a modifier for “training” and explored the concurrent concepts with a graphical literature analysis. The rather complex result is shown in Figure 4.

Figure 3 General landscape of scholarly contributions between the concepts of neurosurgery and medical simulation.

Figure 4 Graphical display of term associations with co-occurrence histories for the concepts of medical simulation, virtual reality and neurosurgery and its subspecialties through 2015.

In 2010, the Congress of Neurological Surgeons (CNS) created a committee called the CNS Simulation Committee. Their goal was to create simulations to maximize resident education and improve outcomes efficiently and safely. In 2012, the CNS created a simulator course that was unveiled at the annual CNS conference. Three main areas were displayed: vascular, cranial and spine. Within each area were courses for medical students, residents, fellows and attendings to participate in. To provide objective data, there was a standardized algorithm for assessment. The participants were given pretests and posttests along with instruction and practice before they were graded for their performance completing certain tasks with each simulation.⁶²

An endovascular course that was described by Fargen et al was put on and involved 37 participants. They were tasked with performing a diagnostic angiogram using a simulator provided by Vist-C Simulator Systems (Mentice, Evanston, IL, USA), Simbionix Systems (Stryker, Kalamazoo, MI, USA) and SimSuite Compass (MicroVention, Tustin, CA, USA). Results showed those who completed the course had a significant improvement in their posttest score and overall performance.⁸² Similar results have been achieved by others for the specific part-task simulations that involved endovascular procedures,^57,83,84 and more comprehensive simulators for the endovascular environment are underway.

The spine courses included a simulator for an anterior cervical discectomy and fusion, which was the first of its kind.⁸⁵ It too followed the same training algorithm as the other modules. Medtronic (Memphis, TN, USA) provided physical models that mimicked the soft tissue of the anterior approach and the bony elements. This allowed participants to practice using Kerrison rongeurs, drills and other real-world tools to increase their familiarity with instrument utilization. This simulator study, although limited in sample size, demonstrated an ability to achieve its goal of improving pretest and posttest scores. There are several other spinal surgery simulators that explore other aspects of spinal surgery and, in particular, complication avoidance in this realm.^{36,69,86–88} When training neurosurgical residents, the training on events that should not happen when providing actual patient care is key and the innovative development of a simulator that teaches the management of cerebrospinal fluid leaks during spinal surgery is one example.⁸⁹

The course put on at CNS shows that there is a large market of different surgical simulators, but they can be mainly categorized into physical and VR simulators. Each has its advantages and disadvantages. Physical models can achieve anatomical accuracy, especially with the engineering tools available today. Three-D printing allows us to create reproducible models that can be subjected to real-world manipulation.⁸ Some physical models are incredibly ornate, such as the traumatic brain injury model used during a course at CNS and the Vascular Simulations Replicator (Stony Brook, NY, USA). They allow real tools and instruments to be used for the learner to become easily accustomed to them and their intended use. However, there are several downsides to physical models. They are generally limited in repetitive use, which in turn requires additional units to be purchased driving up costs. The initial cost of several physical models can be quite daunting. Also, since physical models are mass produced, they often do not demonstrate any anatomic variance or pathologic variance.

When compared to physical models, VR simulators have several advantages. They do not suffer from the issue of repetition since they are digital programs. Their cost can be substantially less as a result.^{28,54,87,90,91} Due to programming and a lack of graphical limitations, VR models can mimic a wide range of anatomic and physical variance. A significant downside to the virtual model is the inability to accurately emulate real tools and instruments so that the user can gain an appreciation for their physical properties and proprioception. This is an evolving field, however, and significant advances have already been made in providing haptic feedback to emulate the real-world properties of tissue and bone.

Simulation in cranial neurosurgery has acquired much attention with the NeuroTouch simulator.^58,72,92 The NeuroTouch training tool was developed by the National Research Council in Canada and is a simulator built around a stereoscope, which incorporates bimanual tools that provide haptic feedback and a real-time computer-generated virtual tissue that reacts to manipulation. By taking advantage of the applied kinesthetic principles, users can simulate real conditions of resecting a lesion using this system. The stereoscope provides binocular vision of the surgical field that conveys depth. The surgical tools include an ultrasonic aspirator, suction, bipolar cautery and microscissors that closely approximate real tools. These tools interface with the real-time virtual tissue to provide sensations to the user that can help differentiate normal brain from tumor and resect it, while also accounting for complications such as bleeding. The argument still holds that it is not identical to real tissue and sensations, but it is a step forward.

This is the only model designed to date that provides computer-generated metrics. The NeuroTouch can generate measurable markers that are categorized into tier 1, tier 2 and advanced tier 2.²⁸ Tier 1 is meant to evaluate safety and quality; metrics are amount of blood loss, tumor resection percentage and brain volume removed. Tier 2 is meant to study motor skills; metrics are total tip path length, maximum force applied, sum of forces utilized, average forces utilized and pedal activation frequency. Advanced tier 2 is used to study complex motor and cognitive bimanual skills interactions; metrics are instrument tips average separation distance, efficiency index, simulated aspirator path length index, coordination index, simulated ultrasonic aspirator bimanual forces ratio. These 13 metrics are designed to assess performance and efficiency.

Data produced here are being used to develop validated levels of achievement to separate levels of training. One study compared the performance of residents and attending neurosurgeons and found significant differences when comparing efficiency index, extent of tumor resection and blood loss, with attending neurosurgeons performing better than residents.⁹³ If these significant findings can be converted into benchmarks or milestones and further validated to differentiate students, junior residents, senior residents and attending neurosurgeons from each other, then there is a great deal of potential for educational applications. This, however, only concludes to establish the difference in skill level using the simulator and not the difference in skill level of handling real tissue or performing surgeries. An attending neurosurgeon may score higher in the simulations due to their real-life experience. A junior learner mastering the simulations may not translate to have matching skill level gained through attending real-life experience. A plateau likely exists, but this has yet to be elucidated. The application of simulations may be limited to early learners.

Cost

It is at best difficult to demonstrate that simulation in neurosurgery can provide real benefits to patients and providers, although it appears intuitive that there is value that benefits patients.^36,88,94 However, little research has been done to evaluate the financial feasibility of implementing a program. In an article by Gasco et al, a cost analysis was performed to determine what the financial burden would be on a program that created such a course.⁹⁵

At the University of Texas Galveston, a simulation program was started for the residents. The department developed a simulation curriculum that offered cadaveric simulations, physical simulator products and haptic feedback simulators. They could institute a curriculum with 68 core exercises per year for each resident for $341,978 to start and $27,876.36 each year to maintain it. These costs include materials, equipment, space and OR time. Costs were mitigated by pursuing industry collaboration, obtaining grants and by using equipment rental programs.⁹⁵

While these numbers are not transferrable to every academic program, they do serve as a framework for future endeavors. A simulation program can be expected to have high startup costs, but yearly upkeep fees are considerably less. Whether this model can be applied toward programs with more than one resident per year is uncertain, but likely feasible. Along with cost, the study analyzed the overall improvement in resident performance, with the greatest strides seen in the more junior groups. This again highlights the utility of simulators in providing a safe environment for novice learners to grow.

Future direction

The future is promising for simulators in neurosurgery as they become more engaging and representative of real surgery. However, their widespread use across neurosurgical training has not yet taken hold. In addition to hurdles such as cost and logistics, simulators still face other obstacles. While several studies have sought to provide validity to both the physical models and VR models, they have not been shown to translate to real surgery. They can distinguish experience based on seniority within the simulator itself. It stands to reason that practicing with NeuroTouch or an anterior cervical discectomy and fusion model before operating on a patient would provide optimal results, but there are no studies investigating this. Large randomized controlled prospective studies are currently of high interest to validate the degree of translation from simulation model to patient. Results obtained from these studies may push for the acceptance of simulators in neurosurgical training for the goal of patient safety.

VR with haptic feedback is providing an immersive experience that is approaching real-world scenarios and situations. Breakthrough training simulation using 3-D printing technology holds promise for future simulation practice, proving high-fidelity patient-specific models to complement residency surgical learning.⁹

Conclusion

Advances in imaging, computing power and interface technology have led to significant interest in and successful development of different simulation models to complement traditional neurosurgical training. Sophisticated VR simulators with some haptic feedback and impressive imaging technology have provided novel options for training in neurosurgery. The current models and simulations at our disposal are less than ideal for real-life translation, but significant strides with haptics have been encouraging. Current studies are assessing improvements, scores and skill levels in learners who use simulation models. The horizon will likely change focus on improvements, scores and skill levels in learners when working on real patients. The application of simulation training in neurosurgery may be limited to early learners. With the support of educational agendas led by national neurosurgical committees, industry and new technology, simulators will become readily available, translatable, affordable and effective.

Disclosure

Dr Schirmer reports honoraria from the American Association of Neurological Surgeons, and ownership stakes in Neurotechnology Investors. The other authors report no conflicts of interest in this work.

References

1.		Guha D, Alotaibi NM, Nguyen N, Gupta S, McFaul C, Yang VXD. Augmented reality in neurosurgery: a review of current concepts and emerging applications. Can J Neurol Sci. 2017;44(3):235–245.
2.		Duffy TP. The flexner report – 100 years later. Yale J Biol Med. 2011;84(3):269–276.
3.		Kohn LT, Corrigan J, Donaldson MS. To Err is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000.
4.		Rajaram R, Chung JW, Cohen ME, et al. Association of the 2011 ACGME resident duty hour reform with postoperative patient outcomes in surgical specialties. J Am Coll Surg. 2015;221(3):748–757.
5.		Taylor DR, Venable GT, Jones GM, et al. Five-year institutional bibliometric profiles for 103 US neurosurgical residency programs. J Neurosurg. 2015;123(3):547–560.
6.		Porter ME. Value-based health care delivery. Ann Surg. 2008;248(4):503–509.
7.		Lee TH, Cosgrove T. Engaging doctors in the health care revolution. Harv Bus Rev. 2014;92(6):104–111, 138.
8.		Clark AD, Barone DG, Candy N, et al. The effect of 3-Dimensional simulation on neurosurgical skill acquisition and surgical performance: a review of the literature. J Surg Educ. 2017.
9.		Weinstock P, Rehder R, Prabhu SP, Forbes PW, Roussin CJ, Cohen AR. Creation of a novel simulator for minimally invasive neurosurgery: fusion of 3D printing and special effects. J Neurosurg Pediatr. 2017:1–9.
10.		Apuzzo ML. The neurosurgeon’s toolbox: from stone age to space age. World Neurosurg. 2011;75(1):1.
11.		Apuzzo ML. The Richard C. Schneider Lecture. New dimensions of neurosurgery in the realm of high technology: possibilities, practicalities, realities. Neurosurgery. 1996;38(4):625–639.
12.		Apuzzo ML. In the realm of ideas: “inside and outside the box”. World Neurosurg. 2011;76(5):363.
13.		Apuzzo ML, Liu CY, Sullivan D, Faccio RA. Surgery of the human cerebrum – a collective modernity. Neurosurgery. 2007;61(1 Suppl):28; discussion 20–31.
14.		Attenello FJ, Lee B, Yu C, Liu CY, Apuzzo ML. Supplementing the neurosurgical virtuoso: evolution of automation from mythology to operating room adjunct. World Neurosurg. 2014;81(5–6):719–729.
15.		Limbrick DD, Dacey RG. Simulation in neurosurgery: possibilities and practicalities: foreword. Neurosurgery. 2013;73(Suppl 4):S1–S3.
16.		Rehder R, Abd-El-Barr M, Hooten K, Weinstock P, Madsen JR, Cohen AR. The role of simulation in neurosurgery. Child’s Nerv Syst. 2016;32(1):43–54.
17.		Singh H, Kalani M, Acosta-Torres S, El Ahmadieh TY, Loya J, Ganju A. History of simulation in medicine: from Resusci Annie to the Ann Myers Medical Center. Neurosurgery. 2013;73(Suppl 1):9–14.
18.		Carter FJ, Schijven MP, Aggarwal R, et al. Consensus guidelines for validation of virtual reality surgical simulators. Simul Healthc. 2006;1(3):171–179.
19.		Bressan F, Buti G, Boncinelli S. Medical simulation in anesthesiology training. Minerva Anestesiol. 2007;73(1–2):1–11.
20.		Pott LM, Budde AO, Murray WB. A proposed classification of simulators. Middle East J Anaesthesiol. 2009;20(2):179–185.
21.		Bova FJ, Rajon DA, Friedman WA, et al. Mixed-reality simulation for neurosurgical procedures. Neurosurgery. 2013;73(Suppl 4):S138–S145.
22.		Gelinas-Phaneuf N, Choudhury N, Al-Habib AR, et al. Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J Comput Assist Radiol Surg. 2014;9(1):1–9.
23.		Hooten KG, Lister JR, Lombard G, et al. Mixed reality ventriculostomy simulation: experience in neurosurgical residency. Clin Neurosurgery. 2014;10 (Suppl 4):576–581.
24.		Sonntag VK. Surgical skills: how can they be assessed? World Neurosurg. 2015;84(5):1201–1202.
25.		Thawani JP, Ramayya AG, Abdullah KG, et al. Resident simulation training in endoscopic endonasal surgery utilizing haptic feedback technology. J Clin Neurosci. 2016;34:112–116.
26.		Pelargos PE, Nagasawa DT, Lagman C, et al. Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery. J Clin Neurosci. 2017;35:1–4.
27.		Davis MC, Can DD, Pindrik J, Rocque BG, Johnston JM. Virtual interactive presence in global surgical education: international collaboration through augmented reality. World Neurosurg. 2016;86:103–111.
28.		Suri A, Patra DP, Meena RK. Simulation in neurosurgery: past, present, and future. Neurol India. 2016;64(3):387–395.
29.		Wu JX, Chen GC, Chang CW, Lin CH. Development of virtual-reality simulator system for minimally invasive surgery (MIS) using fractional-order vascular access. Paper presented at: 2016 SAI Computing Conference, SAI 2016; July 13-15, 2016; London, UK..
30.		Armstrong R, Wright T, de Ribaupierre S, Eagleson R. Augmented reality for neurosurgical guidance: an objective comparison of planning interface modalities. In: Liao H, Zheng G, Lee SL, Cattin P, Jannin P, editors. 7th International Conference on Medical Imaging and Augmented Reality, MIAR 2016. Vol 9805: Springer Verlag; 2016:233–243.
31.		Clarke DB, Kureshi N, Hong M, Sadeghi M, D’Arcy RC. Simulation-based training for burr hole surgery instrument recognition. BMC Med Educ. 2016;16:153.
32.		Singh R, Baby B, Damodaran N, et al. Design and validation of an open-source, partial task trainer for endonasal neuro-endoscopic skills development: Indian experience. World Neurosurg. 2016;86:259–269.
33.		Azarnoush H, Alzhrani G, Winkler-Schwartz A, et al. Neurosurgical virtual reality simulation metrics to assess psychomotor skills during brain tumor resection. Int J Comput Assisted Radiol Surg. 2015;10(5):603–618.
34.		Abla AA, Lawton MT. Three-dimensional hollow intracranial aneurysm models and their potential role for teaching, simulation, and training. World Neurosurg. 2015;83(1):35–36.
35.		Alaraj A, Luciano CJ, Bailey DP, et al. Virtual reality cerebral aneurysm clipping simulation with real-time haptic feedback. Neurosurgery. 2015;11(Suppl 2):52–58.
36.		Bohm PE, Arnold PM. Simulation and resident education in spinal neurosurgery. Surg Neurol Int. 2015;6(1):33.
37.		Regelsberger J, Eicker S, Siasios I, et al. In vivo porcine training model for cranial neurosurgery. Neurosurg Rev. 2015;38(1):157–163.
38.		Shakur SF, Luciano CJ, Kania P, et al. Usefulness of a virtual reality percutaneous trigeminal rhizotomy simulator in neurosurgical training. Clin Neurosurgery. 2015;11(Suppl 3):420–425.
39.		Alotaibi FE, Alzhrani GA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, Del Maestro RF. Neurosurgical assessment of metrics including judgment and dexterity using the virtual reality simulator NeuroTouch (NAJD Metrics). Surg Innov. 2015;22(6):636–642.
40.		Liu Y. Virtual neurosurgical education for image-guided deep brain stimulation neurosurgery. Paper presented at: 4th International Conference on Audio, Language and Image Processing, ICALIP 2014; July 7–9, 2014, Shanghai, China.
41.		Pereira EA, Aziz TZ. Simulation in spinal surgery and the transition from novice to expert. World Neurosurg. 2015;84(6):1511–1512.
42.		Nossek E, Ram Z. Improving vascular neurosurgical skills in an era of diminished microsurgical exposure. World Neurosurg. 2015; 84(4):878–880.
43.		John NW, Phillips NI, Cenydd LA, et al. A tablet-based virtual environment for neurosurgery training. Presence Teleoper Virtual Environ. 2015;24(2):155–162.
44.		Kochan M, Daga P, Burgos N, et al. Simulated field maps for susceptibility artefact correction in interventional MRI. Int J Comput Assist Radiol Surg. 2015;10(9):1405–1416.
45.		Ghobrial GM, Hamade YJ, Bendok BR, Harrop JS. Technology and simulation to improve patient safety. Neurosurg Clin N Am. 2015;26(2):239–243, ix.
46.		Kirkman MA, Muirhead W, Sevdalis N, Nandi D. Simulated ventriculostomy training with conventional neuronavigational equipment used clinically in the operating room: prospective validation study. J Surg Educ. 2015;72(4):704–716.
47.		Rothstein BD, Selman WR. Evaluating simulation as a teaching tool in neurosurgery. Virtual Mentor. 2015;17(1):33–36.
48.		Kirkman MA, Ahmed M, Albert AF, Wilson MH, Nandi D, Sevdalis N. The use of simulation in neurosurgical education and training: a systematic review. J Neurosurg. 2014;121(2):228–246.
49.		Echegaray G, Herrera I, Aguinaga I, Buchart C, Borro D. A Brain Surgery Simulator. IEEE Comput Graphics Appl. 2014;34(3):12–18.
50.		Patel A, Koshy N, Ortega-Barnett J, et al. Neurosurgical tactile discrimination training with haptic-based virtual reality simulation. Neurol Res. 2014;36(12):1035–1039.
51.		Armstrong R, de Ribaupierre S, Eagleson R. A software system for evaluation and training of spatial reasoning and neuroanatomical knowledge in a virtual environment. Comput Methods Programs Biomed. 2014;114(1):29–37.
52.		Shi J. Three-dimensional virtual reality simulation of periarticular tumors using Dextroscope reconstruction and simulated surgery: a preliminary 10-case study. Med Sci Monit. 2014;20:1043–1050.
53.		Wang SS, Li JF, Zhang SM, Jing JJ, Xue L. A virtual reality model of the clivus and surgical simulation via transoral or transnasal route. Int J Clin Exp Med. 2014;7(10):3270–3279.
54.		Zanello M, Zerah M, Sainte-Rose C, Di Rocco F. Virtual simulation in neurosurgery: a comparison between pediatric and general neurosurgeons. Acta Neurochir (Wien). 2014;156(11):2215–2216.
55.		Hochman JB, Sepehri N, Rampersad V, et al. Mixed reality temporal bone surgical dissector: mechanical design. J Otolaryngol Head Neck Surg. 2014;43:23.
56.		Gao B, Guo S, Hu K. Diagnosis and treatment of navigation technology based on the multi-modality image fusion for angioneoplasm. Paper presented at: 11th IEEE International Conference on Mechatronics and Automation, IEEE ICMA 20142014; Tianjin.
57.		Miranpuri AS, Nickele CM, Akture E, Royalty K, Niemann DB. Neuroangiography simulation using a silicone model in the angiography suite improves trainee skills. J Neurointerv Surg. 2014;6(7):561–564.
58.		Choudhury N, Gélinas-Phaneuf N, Delorme S, Del Maestro R. Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg. 2013;80(5):e9–e19.
59.		Alaraj A, Charbel FT, Birk D, et al. Role of cranial and spinal virtual and augmented reality simulation using immersive touch modules in neurosurgical training. Neurosurgery. 2013;72(Suppl 1):A115–A123.
60.		Ferroli P, Tringali G, Acerbi F, et al. Advanced 3-dimensional planning in neurosurgery. Neurosurgery. 2013;72(Suppl 1):A54–A62.
61.		Rosseau G, Bailes J, Del Maestro R, et al. The development of a virtual simulator for training neurosurgeons to perform and perfect endoscopic endonasal transsphenoidal surgery. Neurosurgery. 2013;73(Suppl 4): S85–S93.
62.		Harrop J, Lobel DA, Bendok B, Sharan A, Rezai AR. Developing a neurosurgical simulation-based educational curriculum: an overview. Neurosurgery. 2013;73(Suppl 4):S25–S29.
63.		Cohen AR, Lohani S, Manjila S, Natsupakpong S, Brown N, Cavusoglu MC. Virtual reality simulation: basic concepts and use in endoscopic neurosurgery training. Childs Nerv Syst. 2013;29(8):1235–1244.
64.		Roitberg B, Banerjee P, Luciano C, et al. Sensory and motor skill testing in neurosurgery applicants: a pilot study using a virtual reality haptic neurosurgical simulator. Neurosurgery. 2013;73(Suppl 4):S116–S121.
65.		Stredney D, Rezai AR, Prevedello DM, et al. Translating the simulation of procedural drilling techniques for interactive neurosurgical training. Neurosurgery. 2013;73(Suppl 4):S74–S80.
66.		Jiang D, Hovdebo J, Cabral A, Mora V, Delorme S. Endoscopic third ventriculostomy on a microneurosurgery simulator. Simulation. 2013;89(12):1442–1449.
67.		Matis G, Silva D, Chrysou O, et al. Virtual reality implementation in neurosurgical practice: the “can’t take my eyes off you” effect. Turk Neurosurg. 2013;23(5):690–691.
68.		Harnof S, Hadani M, Ziv A, Berkenstadt H. Simulation-based interpersonal communication skills training for neurosurgical residents. Isr Med Assoc J. 2013;15(9):489–492.
69.		Harrop J, Rezai AR, Hoh DJ, Ghobrial GM, Sharan A. Neurosurgical training with a novel cervical spine simulator: posterior foraminotomy and laminectomy. Neurosurgery. 2013;73(Suppl 1):94–99.
70.		Bambakidis NC, Selman WR, Sloan AE. Surgical rehearsal platform: potential uses in microsurgery. Neurosurgery. 2013;73(Suppl 1):122–126.
71.		Spiotta AM, Rasmussen PA, Masaryk TJ, Benzel EC, Schlenk R. Simulated diagnostic cerebral angiography in neurosurgical training: a pilot program. J Neurointerv Surg. 2013;5(4):376–381.
72.		Delorme S, Laroche D, Diraddo R, F. Del Maestro R. NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training. Neurosurgery. 2012;71(Suppl 1):32–42.
73.		Beier F, Sismanidis E, Stadie A, Schmieder K, Männer R. An aneurysm clipping training module for the neurosurgical training simulator NeuroSim. Paper presented at: Medicine Meets Virtual Reality 19: NextMed, MMVR 20122012; Newport Beach, CA.
74.		Tang HL, Sun HP, Gong Y, et al. Preoperative surgical planning for intracranial meningioma resection by virtual reality. Chin Med J (Engl). 2012;125(11):2057–2061.
75.		Tang K, Li Y, Zhou JA, Zhou Q, Liu C, Zhao YQ. Simulation of petrous bone drilling in subtemporal approach utilizing virtual reality system. Chin J Contemp Neurol Neurosurg. 2012;12(6):736–740.
76.		Gao B, Guo S, Xiao N, Guo J. Development of a 3D blood vessel model for the simulation of the minimally invasive surgery. Paper presented at: 2012 9th IEEE International Conference on Mechatronics and Automation, ICMA 20122012; Chengdu.
77.		Gu SX, Yang DL, Cui DM, et al. Anatomical studies on the temporal bridging veins with Dextroscope and its application in tumor surgery across the middle and posterior fossa. Clin Neurol Neurosurg. 2011;113(10):889–894.
78.		Stadie AT, Kockro RA, Serra L, et al. Neurosurgical craniotomy localization using a virtual reality planning system versus intraoperative image-guided navigation. Int J Comput Assist Radiol Surg. 2011;6(5):565–572.
79.		Shamir RR, Horn M, Blum T, et al. Trajectory planning with Augmented Reality for improved risk assessment in image-guided keyhole neurosurgery. Paper presented at: 2011 8th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, ISBI’112011; Chicago, IL.
80.		Kin T, Oyama H, Shojima M, Shin M, Saito N. Three-dimensional fusion imaging and virtual operative simulation. Jpn J Neurosurg. 2011;20(4):238–246.
81.		Parker DS, Chu WW, Sabb FW, Toga AW, Bilder RM. Literature Mapping with PubAtlas – extending PubMed with a ‘BLASTing interface’. Summit Transl Bioinform. 2009;2009:90–94.
82.		Fargen KM, Arthur AS, Bendok BR, et al. Experience with a simulator-based angiography course for neurosurgical residents: beyond a pilot program. Neurosurgery. 2013;73(Suppl 1):46–50.
83.		Lessard S, Lau C, Chav R, Soulez G, Roy D, de Guise JA. Guidewire tracking during endovascular neurosurgery. Med Eng Phys. 2010;32(8):813–821.
84.		Schirmer CM, Mocco J, Elder JB. Evolving virtual reality simulation in neurosurgery. Neurosurgery. 2013;73(Suppl 1):127–137.
85.		Ray WZ, Ganju A, Harrop JS, Hoh DJ. Developing an anterior cervical diskectomy and fusion simulator for neurosurgical resident training. Neurosurgery. 2013;73(Suppl 1):100–106.
86.		Chou W, Wang T. Human-computer interactive simulation for the training of minimally invasive neurosurgery. Paper presented at: System Security and Assurance2003; Washington, DC.
87.		Gasco J, Patel A, Ortega-Barnett J, et al. Virtual reality spine surgery simulation: an empirical study of its usefulness. Neurol Res. 2014; 36(11):968–973.
88.		Waran V, Narayanan V, Karuppiah R, et al. Injecting realism in surgical training – Initial simulation experience with custom 3D models. J Surg Educ. 2014;71(2):193–197.
89.		Ghobrial GM, Anderson PA, Chitale R, Campbell PG, Lobel DA, Harrop J. Simulated spinal cerebrospinal fluid leak repair: an educational model with didactic and technical components. Neurosurgery. 2013;73(Suppl 1):111–115.
90.		Doyle WK. Low end interactive image-directed neurosurgery: update on rudimentary augmented reality used in epilepsy surgery. Stud Health Technol Informatics. 1996;29:1–11.
91.		Malone HR, Syed ON, Downes MS, D’Ambrosio AL, Quest DO, Kaiser MG. Simulation in neurosurgery: a review of computer-based simulation environments and their surgical applications. Neurosurgery. 2010;67(4):1105–1116.
92.		Alzhrani G, Alotaibi F, Azarnoush H, et al. Proficiency performance benchmarks for removal of simulated brain tumors using a virtual reality simulator neurotouch. J Surg Educ. 2015;72(4):685–696.
93.		Alotaibi FE, AlZhrani GA, Mullah MA, et al. Assessing bimanual performance in brain tumor resection with NeuroTouch, a virtual reality simulator. Neurosurgery. 2015;11:89–98.
94.		Riegel T, Alberti O, Retsch R, Shiratori V, Hellwig D, Bertalanffy H. Relationships of virtual reality neuroendoscopic simulations to actual imaging. Minim Invasive Neurosurg. 2000;43(4):176–180.
95.		Gasco J, Holbrook TJ, Patel A, et al. Neurosurgery simulation in residency training: feasibility, cost, and educational benefit. Neurosurgery. 2013;73(Suppl 1):39–45.