Mixed Reality Improves 3D Visualization and Spatial Awareness of Bone Tumors for Surgical Planning in Orthopaedic Oncology: A Proof of Concept Study

Introduction

CT and MRI are essential preoperative imaging investigations for bone and tumor information for surgical planning in orthopedic oncology. As achieving negative margins in tumor resections determine the oncological outcomes¹, spatial understanding of the patho-anatomy is crucial for the success of tumor resections in various patients with unique anatomies and tumor extents.

Conventionally, tumor surgeons view two-dimensional (2D) medical images in a computer station. Image processing software may create three-dimensional (3D) models from 2D imaging datasets. Visualizing virtual 3D models on the 2D flat screen of the computer station lacks depth perception and parallax compared to physical 3D models. When surgeons clinically examine bone tumor patients before the surgeries, they must mentally process and correctly overlay the patients’ virtual 2D medical images and 3D bone-tumor models onto their anatomy. Also, surgeons must mentally retain the spatial orientation of the matched virtual 3D models when viewing patients from different angles. Appropriate surgical exposures and the osteotomies along desired planes are then decided. The visual-spatial ability (VSA) to mentally visualize and manipulate objects in 3D space is highly demanding,² especially when the amount of simultaneously processed information is substantial for large tumors, regions with complex anatomies, and bone geometries.

Computer navigation and 3D-printed resection guides facilitate precise osteotomies^3,4 but mainly benefit orthopaedic procedures after surgical exposure. Computer navigation allows tracking of the diseased bones with reference to the preoperative medical images after placing a navigation tracker into the patient’s bone. Also, surgeons operate by viewing the 2D images in the navigation display and are distracted from the operative fields.⁵ Although physical 3D-printed models may provide a viable alternative for visualizing the patient’s patho-anatomy with tactile feedback,³ mental translation of the spatial relationship of physical 3D models to the actual patients’ anatomy is still required. Both technologies may not overcome the surgeons’ cognitive load in the spatial understanding of bone tumors in actual patients before the surgery starts.

Mixed Reality (MR) is an immersive technology with spatial computing in merging real and virtual worlds, and users can interact with digital objects in real time.⁶ The virtual digital objects in MR are 3D holograms that retain the depth and parallax perception of the original physical objects. Through Head-Mounted Displays (HMDs), surgeons stereoscopically view the 3D holograms from 360° in their physical environment. Patient-specific 3D holograms are generated from 2D medical images. By superimposing the 3D holograms on the patient, surgeons can intuitively understand the spatial relationship of bone tumors to the patient’s anatomy. Clinical reports of MR application are limited, with only a few on spine pedicle screw placement⁷ and shoulder arthroplasty.⁸ The data in orthopaedic oncology is lacking. In this study, we evaluated 1) surgeons’ spatial awareness of tumor locations and orientations by viewing patient-specific 3D holograms; 2) the cognitive load of surgeons when using MR technology for preoperative clinical assessment of bone tumor patients.

Methods

The study was performed under the ethical standards of the Joint Chinese University of Hong Kong-New Territories East Cluster Clinical Research Ethics Committee (the Joint CUHK-NTEC CREC) (CRE Ref. No. 2022.543) of the authors’ hospital. Informed verbal consent was obtained from the patients participating in the study. The verbal consent instead of written consent was approved by the Joint CUHK-NTEC CREC. The study complied with the Declaration of Helsinki. Between July 2021 and December 2022, we retrospectively reviewed nine patients with primary bone tumors having surgery. There were six primary bone sarcomas, two benign bone tumors, and one revision pelvic prosthesis (Table 1). The tumor locations were pelvis (three), tibia (two), proximal femur (one), scapula (one), proximal humerus (one), and calcaneus (one). All patients (except Patient 1) underwent bone resections with five patients under the assistance of 3D-printed resection guides. Three patients had 3D-printed patient-specific implants for bone reconstruction, one with iliac crest bone graft, two with cementation, and no reconstruction in two patients. Surgeons used MR technology for preoperative clinical assessment. MR technology was thought necessary for preoperative clinical evaluation of bone tumor patients because of anticipated difficulties in spatial understanding of complex anatomical areas such as the pelvis, intraosseous tumor involvement, sites of planned osteotomies, or the placement of patient-specific guides and implants (Figure 1A–H).

Table 1 Demographic Information of the Patients Using Mixed Reality Technology in the Study

Figure 1 The axial (A) and coronal (B) views of CT images in Patient 1 with giant cell tumor involving femoral head and neck (red arrows). The axial view (C) of T1-weighted MRI with contrast and 3D surgical planning (D) with patient-specific cutting guides (yellow arrows) in Patient 3 with left scapular osteosarcoma (red arrows). The coronal view (E) of T1-weighted MRI and axial view (F) of Proton Density sequence MRI in Patient 4 with left acetabular low-grade chondrosarcoma (red arrows). The coronal view (G) of T2 weighted MRI in Patient 9 with tibia osteosarcoma (red arrows) and 3D surgical planning (H) with patient-specific cutting guides (red arrows).

Preparation of MR Holographic Application

The clinical workflow of mixed Reality in orthopedic oncology has been described⁹ (Figure 2). The MR software platform was developed (Syngular Technology Limited, Hong Kong SAR, China), and MR holographic application was prepared for each case. CT ± MRI images were acquired as both are essential preoperative imaging investigations for the workup of musculoskeletal tumors. The 2D medical images in DICOM format were obtained for further processing. 3D bone models were generated from CT images as bones have a high contrast signal. MRI images better show soft tissue structures, and medullary and extraosseous tumor volumes were mapped. 3D bone-tumor models were then created with their geometries represented in polygons. Polygon-based Computer-Aided Design (CAD) models were thus produced from 2D medical images. Computer Graphics software generates cinematic-rendered (CR) models with edited color, material, texture, and the appropriate illumination to simulate the natural way light would reflect off of physical objects in the real world. The processing gives CR models a more photorealistic depiction of the 3D models (Figure 3A–E). A 3D Engine (Unity Technologies, Unity Software Inc., San Francisco, US), a system commonly used for virtual computer simulations, was then used to develop the User Interface (UI) for the holographic contents, including 2D medical images, digital medical information, and 3D models. The holographic application calculates the visual appearance of the scene, and the interactive 3D holograms were then created. Its content was exported into a digital package and loaded into the MR-HMD (Hololens 2, Microsoft Corporation, Redmond, WA, US).

Figure 2 Shows the workflow of generating the mixed reality software application used for surgical planning of bone tumors. Abbreviations: DICOM, Digital Imaging and Communications in Medicine; CT, Computer Tomography; MRI, Magnetic Resonance Imaging; CAD, Computer-Aided Design; UI, User Interface; MR, Mixed Reality; HMD, Head-Mounted Display.

Figure 3 (A) shows cinematic-rendered 3D holograms of the pelvis and proximal femur in Patient 6 with the osteochondroma of the right ilium (red arrow). Using hand gestures, the surgeon interacts with the 3D holograms and the 2D image of the CT pelvis (yellow arrow). The whole image datasets in axial, reformatted coronal, and sagittal views can be viewed and scrolled through in real-time. (B) shows the superior and posterior views of the pelvic and Hip 3D holograms in Patient 6 with osteochondroma of the right ilium (red arrows) and proximal femur (yellow arrow). (C) After matching the cinematic-rendered 3D hologram of the tibia osteosarcoma to the leg of Patient 9, the surgeon examines by visualizing the osteosarcoma (red arrows) with an improved spatial understanding of tumor boundaries. (D) The resection planes and patient-specific cutting guides (red arrows) can also be integrated into the holograms that facilitate the resection planning and positioning of the cutting guides. (E) The bone reconstruction with a patient-specific implant and its respective drill guide for screw fixation (red arrow) can be visualized. The resected tibia bone hologram enables the surgeon to mark the osteotomy site (yellow arrow) precisely before the skin incision. The mixed reality hologram visualization on patents enhances surgeons’ capacity for complex surgical planning.

Surgical Planning and Clinical Examination

The surgeon (KCW) performed the surgical planning by clinically accessing each bone tumor patient using conventional 2D and MR 3D hologram methods. In the conventional 2D method, the surgeon reviewed the 2D image data, planning information of 3D-printed guides and patient-specific implants (if any), then mentally overlaid the virtual 3D models onto the patient’s body (Figures 4A, B, D–F; 5A–C, E and F). In the MR 3D hologram method, the surgeon visualized the 3D holograms directly on the patients via HMD. The HMD contains cameras that track the hands of the surgeons. Hand gestures were used for real-time interaction with the 3D holograms. Surgeons manually matched the holograms onto the patient’s anatomy. The registration was verified as correct by palpating the nearby bony landmarks in pelvis (anterior superior iliac spine, posterior superior iliac spine or pubic tubercle); tibia (medial and lateral malleoli or tibial tuberosity); humerus/scapula (greater tuberosity, clavicle, acromion or scapular spine); calcaneus (calcaneal tuberosity). Then, surgeons “saw-through” the patient’s body with superimposed 2D medical images and 3D models (Figure 4C and G, Supplementary Videos 1 and 2). The surgical incision, exposure, sites of osteotomies, 3D-printed guides, and patient-specific implants were then determined while clinically accessing each patient with both methods (Figure 5D, G and H, Supplementary Videos 3 and 4).

Figure 4 Patient 5 had left pelvic low-grade bone sarcoma with resection and saddle prosthesis reconstruction in 2005. He presented with increasing back pain and ipsilateral knee pain due to limited hip motion. (A) shows the plain radiograph of the pelvis with a saddle prosthesis with pseudo articulation at a higher hip center. Bone overgrows around the saddle component (red arrow). (B) shows the surgical planning of bone resection at the ilium and reconstruction of a patient-specific implant with standard hip arthroplasty at the near normal hip center. (C) By visualizing the 3D hologram on Patient 5 with distorted pelvic anatomy like X-ray vision, the surgeon “sees through” the skin and determines the location of the old implant, the surgical approach, and the skin incision site in planning the revision surgery (Supplementary Video 1). Patient 8 had right posterior proximal tibia parosteal osteosarcoma with a large extraosseous tumor component (red arrows) shown on the axial (D) and sagittal (E) views of MRI images. (F) shows the surgical planning of the hemicortical tumor resection in the 3D bone tumor model with planned resection planes (red arrows). (G) As MRI images better delineate the tumor extent (red arrows) and soft tissue structures, the mixed reality feature of superimposing MRI image dataset further increases the surgeon’s ability to understand the spatial relationship of bone tumors for surgical planning before skin incision (Supplementary Video 2).

Figure 5 The axial (A) and sagittal (B) views of CT images show recurrent low-grade chondrosarcoma involving the posterior proximal humerus (red arrows) in Patient 7. (C) shows the 3D surgical planning of the hemicortical tumor resection assisted by a patient-specific cutting guide (red arrows). (D) Viewing the overlaid 3D holograms (bone and cutting guide) on Patient 7 allows easy marking of skin incisions and provides better spatial awareness of the cutting guide placement (red arrows) (Supplementary Video 3). (E) shows the 3D surgical planning of partial acetabular resection assisted by a patient-specific cutting guide (red arrows) in Patient 4 with low-grade chondrosarcoma. (F) shows the planning of reconstruction of the acetabular defect with a patient-specific implant (red arrows) in the same Patient 4. A dual mobility cup was cemented into the acetabular cup (black arrow) of the pelvic implant. 3D holograms of patient-specific cutting guides (G) (red arrows) and implants (H) (red arrows) could be visualized for surgical planning before skin incisions (Supplementary Video 4).

No quantitative tool was available to assess the users’ experience in spatial awareness of bone tumors during preoperative assessment. Therefore, for each method, the surgeon completed a qualitative survey 1) a Likert-Scale (LS) questionnaire to assess his opinions on the spatial awareness of the bone tumors and the effectiveness of surgical planning and 2) The National Aeronautics and Space Administration-Task Load Index (NASA-TLX) to commute the surgeons’ subjective cognitive workload.^10,11 The LS questionnaire uses five-point rating scales to assess the surgeons on five specific domains: spatial visualization, spatial orientation, depth perception, identification of the location of the pathology, ease of determining surgical incision, and effectiveness as a preoperative assessment tool. Spatial visualization is the ability to look at a 2D figure and visualize what it would look like expanded into 3D, then mentally rotate and manipulate objects without the physical objects in front of you.¹² Spatial orientation is the ability to identify objects’ position or direction in space mentally.¹³ The LS questionnaire was modified from the previous study by Lu et al.¹⁰ It had a five-point scale where “1” relates to a negative experience, and “5” is the maximum degree of satisfaction. NASA-TLX is a measuring tool to determine an overall cognitive workload rating after the surgeon performs the task. The surgeon rated his score on an interval scale between 0 and 100 on six subscales.^14,15 The higher score indicates a more significant workload of the respective subscales. The six parameters are 1) Mental demand - how much thinking, deciding, or calculating was required?; 2) Physical demand - The amount and intensity of physical activity required; 3) Temporal demand – the amount of time pressure involved; 4) Effort - how hard did you work to maintain your level of performance?), 5) Performance – the level of success in completing the task, and 6) Frustration level - How insecure, discouraged, or secure or content you felt during the task.

All resected bone tumor specimens were examined histologically, and six patients with bone sarcoma were also assessed for surgical margins.

Statistical Analysis

The results of the LS questionnaire and NASA-TLX score using conventional 2D and MR 3D hologram methods for preoperative surgical planning in bone tumor patients were recorded and compared. The results were summarized in the median and interquartile range (IQR). The scores of two methods (2D vs MR 3D hologram) in each domain were visually compared using boxplots. Data analyses were conducted using R 4.2.1 (R Foundation for Statistical Computing, Vienna, Austria).

Results

The qualitative survey based on the LS questionnaire revealed that the MR 3D hologram group seemed to have superior results in all domains of spatial awareness of tumors and was considered more effective as a preoperative planning tool than the conventional 2D Group (Table 2, Figure 6A). For NASA-TLX scores, the overall cognitive workload was lower in MR 3D hologram group than in the 2D Group for preoperative clinical assessment (2D group median score 75, IQR 21.25–68.75 versus MR 3D group median 17.5, IQR 20–42.5). It is evident that “Mental Demand”, “Effort”, and “Frustration” were the dominant source of cognitive workload in the conventional 2D Group, while “Physical Demand” was in the MR 3D hologram group (Table 2, Figure 6B). When using MR technology with HMDs, the surgeon reported no discomfort, like motion sickness, headache, fatigue, or eye strain.

Table 2 Results of Likert Scale Questionnaire Scores and NASA Task Load Index Scores

Figure 6 Shows the qualitative results of a Likert-Scale (LS) questionnaire (A) on the spatial awareness of bone tumors and the effectiveness of surgical planning and the National Aeronautics and Space Administration-Task Load Index (NASA-TLX) score (B) on the surgeon’s subjective cognitive workload using mixed reality technology. Abbreviations: 2D, Conventional method of viewing two-dimensional medical images and planning information; MR, surgical planning by viewing three-dimensional holograms that were superimposed on the patients.

Histological examination of the resected specimens showed that two of the six bone sarcoma patients had positive microscopic margins in the soft tissue. All bone resection margins were negative.

Discussion

Surgical planning is challenging in complex bone tumor surgery. Surgeons must mentally integrate preoperative 2D images and superimpose virtual 3D models onto the patient’s anatomy to determine the surgical approach and osteotomies around the bone tumors. Although computer navigation and 3D-printed guides enable precise osteotomies, the technologies are only feasible after surgical exposure. Our preliminary result suggested that when compared with the conventional 2D method, MR 3D holograms may improve 3D visualization and spatial awareness of bone tumors in the patient’s anatomy without an increase in cognitive workload during the preoperative surgical assessment. The new technology may facilitate surgical planning before skin incisions in orthopaedic oncology surgery.

The study has several limitations. 1) It only included a small sample size (nine cases) that did not allow for meaningful statistical analysis. A larger number of patients is needed to conclude the statistical significance of potential superior results in mixed reality technology. 2) It is the experience of a single orthopaedic oncology surgeon using mixed reality technology. Multiple surgeons are required for comparison to reduce assessors’ bias in future studies. 3) Improved spatial awareness of the tumor locations or orientations in preoperative clinical assessment using MR may not translate into better clinical outcomes governed by other factors like tumor grades, response to chemotherapy, availability of assistive tools to replicate surgical plans intraoperatively, or resection margins. 4) Mixed reality technology is still developing in the orthopaedic field. Institutes may need help accessing the technology, and no mature software platform is dedicated to orthopaedic oncology. 5) The LS questionnaire and NASA-TLX are subjective qualitative methods of assessing new technology. Although the generic subscales allow the index to be used across multiple domains, the index was rated post-task, and the surgeon may need to remember the details. However, the assessment methods provide a quick and straightforward estimate of surgeons’ experience with the MR technology about its effectiveness and cognitive workload in surgical planning. The result of the study may be a proof-of-concept for MR application in orthopaedic oncology.

The study was the first case series using MR technology to assess the surgeons’ spatial awareness of bone tumors in patients’ anatomies during preoperative assessment in orthopaedic oncology. The results of the LS questionnaire suggested that overlaying the patients’ anatomies with patient-specific 3D holograms enhanced surgeons’ spatial awareness of tumors’ locations and orientations. The results concurred with the findings when the MR technology was applied to tumors inside the visceral organs during open abdominal surgery,¹⁶ preoperative assessment in cervical spine fracture,¹⁷ and tibia fracture.¹⁸ The real-time access to patient-specific 3D holograms during patients’ clinical examinations greatly facilitates surgical planning. It may enhance surgeons’ ability and reduce the learning curve in complex surgical planning.¹⁹ With MR technology, surgeons can simulate the planned operations with the improved spatial awareness of tumors before the actual surgeries in the operation rooms. It may help achieve more precise skin incisions and have better access to the underlying tumors. The potential benefit may improve surgical accuracy, safety, and efficiency.^17,20,21

MR 3D hologram group showed a significantly lower NASA-TLX score with less “mental demand”, “effort”, “frustration”, and better “performance” than the conventional 2D Group. With patients’ anatomies augmented with 3D holograms during clinical assessment, MR was perceived as less cognitively demanding and may lead to higher performance. Our results are similar to the case series of using MR for the preoperative assessment of one hip fracture, two spine fractures, and one pelvic bone sarcoma.¹⁰ In the study, one distinct advantage of MR holograms was that they preserved the actual size of the original models, and the real-time interaction with CR holograms provided clear depth cues. The features can only be experienced in 3D-printed physical models but are absent when viewing the same 3D objects on a 2D computer display. The overlaying holograms of patient-specific guides and implants on patients’ anatomies may further enhance the ease of surgical planning before skin incisions.

With less cognitive load and improved ergonomics on wearing MR HMD, surgeons can stay focused on the patients and surgical tasks while keeping their hands free without losing the sterility of the surgical field in the operating room.⁹ Therefore, MR technology addresses the limitations of attention shift in computer navigation⁵ or absent real-time image feedback in 3D printed guides²² for orthopaedic oncology. The various assistive tools may be complementary in improving surgical care but are not mutually exclusive in clinical applications, as each has its inherent strengths and weakness. Surgeons may choose the tools most suitable for their patients; based on the tumor characteristics, reconstructive options, and available facilities and expertise.

Two of the six sarcoma patients had positive resection margins in the soft tissue, while all bone resection margins were negative. As soft tissue anatomy distorted after skin incision and differed from preoperative imaging, surgeons still adopted a conventional technique in soft tissue resection. Therefore, the MR technology may facilitate bone resection after surgical exposure, similar to computer navigation²³ and 3D printed guides.

Further studies can investigate whether improving preoperative spatial awareness of tumors’ locations and orientations using MR technology can translate into better oncological and functional outcomes. The exact clinical roles of other MR features, like mobile computing with instant access to medical data, remote assistance, intraoperative guided osteotomies, and surgical training and education, should also be explored.

Conclusion

Our study suggested that the spatial computing in MR technology may improve 3D visualization and spatial awareness of bone tumors in patients’ anatomies and may facilitate surgical planning before skin incisions in orthopaedic oncology surgery. It potentially complements the limitations of attention shift in computer navigation or lacking real-time image feedback in 3D printed guides. It may improve patient care and outcomes. Further studies are needed to investigate whether MR technology improves clinical outcomes.