Creating a Digital Twin of Spinal Surgery: A Proof of Concept

Abstract: Surgery digitalization is the process of creating a virtual replica of real-world surgery, also referred to as a surgical digital twin (SDT). It has significant applications in various fields such as education and training, surgical planning, and automation of surgical tasks. In addition, SDTs are an ideal foundation for machine learning methods, enabling the automatic generation of training data. In this paper, we present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. The proposed digitalization focuses on the acquisition and modelling of the geometry and appearance of the entire surgical scene. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion. We justify the proposed methodology, discuss the challenges faced and further extensions of our prototype. While our PoC partially relies on manual data curation, its high quality and great potential motivate the development of automated methods for the creation of SDTs.

• The paper introduces a high-fidelity surgical digital twin for spinal surgery by integrating laser scanning and RGB-D imaging.
• It utilizes advanced data acquisition and fusion methods to capture both static OR features and dynamic elements like surgeon movements.
• Quantitative results indicate sub-millimeter accuracy and near-perfect alignment with real camera feeds, highlighting potential for enhanced surgical training and automation.

Creating a Digital Twin of Spinal Surgery: A Proof of Concept

Introduction

The paper explores the development of a surgical digital twin (SDT) focusing on the digitalization of spinal surgery procedures, specifically pedicle screw placement, performed in a controlled laboratory setting. The SDT serves as a comprehensive virtual replica of the operating room (OR), providing detailed geometric and semantic data crucial for visualization, analysis, and potential robotic or ML-driven task automation.

Methodology

Data Acquisition and Fusion

The methodology includes capturing static and dynamic elements within the OR using a range of technologies. Laser scanners generate a foundational 3D point cloud to represent static environment features, such as the OR infrastructure, lighting, and fixed equipment.

Figure 1: Generation of the reference point cloud from multiple laser scans, illustrating the successful fusion from individual scans with minimized occlusion challenges.

Dynamic and Static Element Modelling

The paper employs multiple imaging technologies to accommodate the dynamic nature of surgery— RGB-D cameras were used for capturing body poses and instrument movements. Additionally, dedicated photogrammetry was applied to construct highly accurate models of the surgical tables, tools, and anatomical components before integrating these into a unified virtual environment.

Human Pose and Instrument Tracking

Human poses are estimated using a blend of RGB-D input data and machine learning algorithms utilizing the SMPL-H model alongside OpenPose. This setup allows for precise, marker-less tracking of the surgeon’s movements throughout the procedure.

Results Evaluation

Quantitative Assessment

The system demonstrated high geometric accuracy in both static and dynamic scenarios, with laser scan fusion achieving sub-millimeter errors and RGB-D camera calibration errors averaging below 1 pixel, confirming the precision of the SDT’s spatial data amassment.

Qualitative Outputs

Rendering results compared the SDT against real-world camera feeds, with near-perfect alignment evidenced in Figures 4 and 5, underscoring the fidelity of the digital models.

Figure 2: Comparisons between the digital twin renders and real camera images, reflecting perspectives from various Kinect cameras.

Figure 3: Renderings depicting the reconstructed operating room and estimated body poses, exhibiting the prototype's rich visual details.

Discussion

Technical Challenges and Limitations

The primary challenge arises from reflective and dynamic surfaces within the OR, which impose significant complexities for optical data acquisition systems. Moreover, while the paper makes substantial strides in static and dynamic tracking, integration with real-time intraoperative imaging or real-time anatomy deformation remains unexplored due to existing technological limitations in reconstructing dynamic, deformable anatomy during actual surgical procedures.

Future Directions

The successful realization of an SDT opens avenues for broader digital twin applications. This involves potential advancements in surgical training, real-time task automation, and simulation-based preoperative planning. There remains a clear pathway towards enhanced integration of sensor modalities, inclusion of more sophisticated AI-driven predictive analytics, and automation techniques to further refine and extend the current SDT model.

Conclusion

The research demonstrates the feasibility and practicality of developing a digital twin for surgeries by utilizing a combination of high-fidelity image capturing, dynamic modeling, and integrated data processing. The SDT establishes a platform for future explorations into educational and operational enhancements in surgical contexts. Such advancements will drive the evolution of precision surgery, provide robust platforms for robotics and AI applications in medical fields, and offer substantial improvements to surgical outcomes and preemptive problem-solving.