Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging

Abstract: The use of digitally reconstructed radiographs (DRRs) to solve inverse problems such as slice-to-volume registration and 3D reconstruction is well-studied in preoperative settings. In intraoperative imaging, the utility of DRRs is limited by the challenges in generating them in real-time and supporting optimization procedures that rely on repeated DRR synthesis. While immense progress has been made in accelerating the generation of DRRs through algorithmic refinements and GPU implementations, DRR-based optimization remains slow because most DRR generators do not offer a straightforward way to obtain gradients with respect to the imaging parameters. To make DRRs interoperable with gradient-based optimization and deep learning frameworks, we have reformulated Siddon's method, the most popular ray-tracing algorithm used in DRR generation, as a series of vectorized tensor operations. We implemented this vectorized version of Siddon's method in PyTorch, taking advantage of the library's strong automatic differentiation engine to make this DRR generator fully differentiable with respect to its parameters. Additionally, using GPU-accelerated tensor computation enables our vectorized implementation to achieve rendering speeds equivalent to state-of-the-art DRR generators implemented in CUDA and C++. We illustrate the resulting method in the context of slice-to-volume registration. Moreover, our simulations suggest that the loss landscapes for the slice-to-volume registration problem are convex in the neighborhood of the optimal solution, and gradient-based registration promises a much faster solution than prevailing gradient-free optimization strategies. The proposed DRR generator enables fast computer vision algorithms to support image guidance in minimally invasive procedures. Our implementation is publically available at https://github.com/v715/DiffDRR.

• The paper introduces a fully differentiable DRR generation framework that leverages vectorized tensor operations in PyTorch for real-time gradient-based optimization.
• The methodology vectorizes Siddon’s method to achieve auto-differentiation and significant runtime improvements over traditional CPU-based techniques.
• Utilizing the auto-differentiable approach, the method enhances image registration applications, supporting rapid inverse problem-solving for intraoperative imaging.

Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging

Introduction

Digitally reconstructed radiographs (DRRs) simulate 2D X-ray images from 3D computed tomography (CT) volumes using ray-tracing techniques. While essential in preoperative settings like radiation oncology, DRRs face constraints in intraoperative imaging due to real-time generation challenges. Algorithmic enhancements and GPU implementations have accelerated DRR creation, yet optimization via DRRs remains sluggish due to the lack of gradient availability in most DRR generators. By reformulating Siddon's method—the prevalent ray-tracing technique for DRR generation—into vectorized tensor operations, it becomes possible to execute DRR generation within PyTorch's framework, leveraging its auto-differentiation for gradient-based optimization and compatibility with deep learning.

Figure 1: DRR synthesis. (a) Idealized model of a projectional radiography system; (b) Siddon's method ray tracing; (c) Vectorized performance improvements.

Methodology

Siddon's Method Verklarung

The classical approach to generating DRRs employs Siddon's method, a parametric strategy for identifying the intersections of a ray with orthogonal planes in a voxel space [Siddon, 1985]. Equation-based integrations account for X-ray energy attenuation across these intersections, parameterized by $\alpha$. The iterative nature of Siddon-Jacobs' variant [Jacobs et al., 1998] limits vectorization potential, prompting the development of a fully vectorized implementation.

Introduction to Vectorization

The proposed method reformulates Siddon's approach using vectorized tensor operations within PyTorch, parallelizing computation across image plane coordinates and ray intersections. This reformulation utilizes the GPU to expedite tasks and integrate automatic differentiation, making the resultant DRR both swift and versatile in optimization contexts.

Derivative Framework

A critical innovation is achieving differentiability in DRR generation with respect to imaging parameters such as source-detector geometry and orientation. PyTorch’s automatic differentiation capabilities allow seamless gradient extraction for any differentiable loss function, thus enabling DRR application in optimization models for inverse problem-solving in medical imaging.

Experiments and Results

Performance Comparative Analysis

The novel vectorized method (VGS) was benchmarked against CPU-based implementations and a customized Siddon-Jacobs GPU version (VGSJ), demonstrating runtime parity with state-of-the-art GPU accelerators and significantly outperforming conventional CPU-based tools (Table 1).

Table 1: Benchmark results for timing and accuracy metrics among different implementations. | %%%%1%%%% | VGS Timing (ms) | VGSJ Timing (ms) | CP Timing (ms) | RMSE VGS vs CP | Autograd vs FFD | |-------|----------------|------------------|---------------|---------------|-----------------| | 100 | 17.6 ± 0.05 | 380 ± 20.9 | 1028 ± 132 | (6.9±2.2)×10⁻⁴ | 0.03 ± 0.02 |

Registration Applications

The algorithm was applied to gradient-based slice-to-volume registration using DRRs, demonstrating superiority over gradient-free methods by leveraging the convex nature of zero-normalized cross-correlation (ZNCC) landscapes around the optimum (Figure 2).

Figure 2: Convexity demonstration of negative ZNCC around optimal DRR generating parameters.

Figure 3: Differentiable DRRs executing slice-to-volume registration highlighting convergence efficacy.

Discussion

The presented DRR generation model offers an effective tool for addressing inverse problems in intraoperative imaging through gradient-based techniques impossible with previous models. This advancement opens pathways to rapid, real-time image-guided surgery applications, where swift and robust optimization is crucial. Future efforts involve integrating these advancements with machine learning frameworks for advanced imaging tasks, potentially coupling with methods like Monte Carlo-based scattering estimations to enhance realism and application scope.

Conclusion

This development in digitally reconstructing radiographs via auto-differentiable vectorized techniques stands to significantly impact precision in intraoperative imaging, simplifying optimization workflows with efficient, scalable gradient-based approaches. Future explorations may reveal deeper synergies with deep learning potentials, heralding broader applicability across medical imaging domains.