Point-based Differentiable Rendering

• Point-based Differentiable Rendering (PBDR) is a framework that models scenes as discrete, learnable points with attributes and uses differentiable splatting for image synthesis.
• It employs continuous, piecewise-smooth functions enabling gradient-driven optimization of geometry, appearance, and camera parameters through backpropagation.
• PBDR scales from simple point clouds to hundreds of millions of points, finding use in inverse rendering, novel-view synthesis, and large-scale scene reconstruction.

Point-based Differentiable Rendering (PBDR) is a computational framework that renders and optimizes 3D point-based scene representations using differentiable splatting and compositing methods. In contrast to volumetric rendering approaches, PBDR directly parameterizes scene geometry (and sometimes reflectance) as a set of discrete, learnable primitives—typically points augmented with radii, color, and possibly higher-order appearance coefficients. The splatting-based rasterization schemes of PBDR guarantee differentiability with respect to point parameters, enabling gradient-driven optimization from image-based objectives. PBDR spans applications in inverse rendering, radiance field synthesis, surface denoising, point cloud upsampling, and large-scale scene reconstruction.

1. Mathematical Foundations of PBDR

PBDR models a scene as a collection of points $\{\mathbf{x}_i\}$, each equipped with per-point attributes (e.g., radii, normals, BRDF or spherical harmonics coefficients, color, opacity). Each point projects to the image plane under a camera model, and its contribution to a pixel is computed by a "splat"—most often a (possibly anisotropic) Gaussian, ellipse, or soft indicator function.

A canonical form of the rendered pixel color $I(u)$ is

$$I(u) = \sum_{i=1}^N R_i \, \alpha_i(u) \prod_{j < i}[1 - \alpha_j(u)]$$

where $R_i$ is the radiance attributed to point $i$, $\alpha_i(u)$ is a soft-occupancy ("opacity") function, and the product ensures proper front-to-back compositing analogous to the Porter–Duff over operator. For example, in DPIR (Chung et al., 2023), $\alpha_i(u)=1-\exp\left(-\|p_i-u\|^2/r_i^2\right)$ with quadratic or exponential falloff.

Other notable variants include elliptical weighted average splatting (Kopanas et al., 2021), anisotropic Gaussian splats (Yifan et al., 2019), fixed-radius RBFs (Zhang et al., 2022), or one-pixel splats with deep-shading completion (Rückert et al., 2021). The splatting function is selected for both rendering fidelity and gradient stability.

Scene appearance in PBDR can be modeled per-point via spatially-varying BRDF MLPs (Chung et al., 2023), spherical harmonics (Zhang et al., 2022), or learned feature embeddings as in neural point-based renderers (Kopanas et al., 2021).

2. Differentiability and Gradient Backpropagation

The differentiability of PBDR is rooted in its splatting-based rasterization, in which all steps are composed of continuous, piecewise-smooth functions:

• Point projection to screen space (Jacobian of camera model),
• Splats as exponential or sigmoid kernels (closed-form gradients with respect to point position, size, and appearance),
• Soft visibility (via alpha-compositing, probabilistic or relaxed depth tests for differentiability),
• Mixing via front-to-back compositing or weighted averaging.

For example, the gradient of the rendered image with respect to point radiance is

$$\frac{\partial I(u)}{\partial R_k} = \alpha_k(u) \prod_{j < k} (1-\alpha_j(u))$$

while gradients with respect to position and radius derive via the chain rule through $\alpha_k$ and the projection function.

Occlusion handling is implemented through soft approximations of depth-sorting—e.g., sigmoid relaxations, probabilistic depth tests over splat distributions (Kopanas et al., 2021), or continuous shadow mapping as in DPIR (Chung et al., 2023), thus allowing analytic derivatives.

End-to-end, all parameters—including geometry, appearance, camera intrinsics, extrinsics, photometric pipeline coefficients, and even neural environment maps—can be optimized by backpropagation. Modern frameworks leverage batched GPU tensor kernels for this entire chain (Zhao et al., 23 Dec 2025), enabling large-scale optimization.

3. Scene Representations and Extensions

PBDR supports a variety of point-based parameterizations, balancing efficiency, memory, and reconstruction fidelity:

• Simple Point Clouds: Each point is parameterized by 3D position, color, radius, and possibly opacity. Used in radiance field PBDR (Zhang et al., 2022) and large-scale scene models (Zhao et al., 23 Dec 2025).
• SDF-Augmented Points: Hybrid of discrete points and an implicit signed distance function, as in DPIR (Chung et al., 2023), to provide both high-fidelity surface normal estimation and splatting stability.
• High-order Appearance Encoding: Per-point spherical harmonics (Zhang et al., 2022), BRDF basis expansions (Chung et al., 2023), or learned neural features (Kopanas et al., 2021, Rückert et al., 2021).
• Temporal/Dynamic Scenes: Extension to 4D (3D+time) splats, as in Gaian (Zhao et al., 23 Dec 2025), and per-frame point clouds with temporal warm-starting (Zhang et al., 2022).
• Self-supervised upsampling: Self-supervision via image-space consistency, with differentiable silhouette rendering enforcing geometric and visual alignment (Zhao et al., 2021).

The representations are initialized from visual hull estimation, MVS, or random sampling, and are refined via point pruning, upsampling, or coarse-to-fine schemes for optimal coverage and optimization (Zhang et al., 2022).

4. Inverse Rendering and Optimization Landscapes

PBDR is particularly effective for inverse problems, such as shape and reflectance estimation from multi-view and multi-light image collections. The standard workflow includes:

• Initialization: Uniform or mask-filtered points seeded within visual hulls.
• Rendering: Project and splat points with differentiable compositing.
• Objective: Photometric losses on rendered vs. ground-truth images (e.g., $\|I-I'\|_2^2$), optionally augmented with SSIM, mask, and appearance regularizers; for reflectance, basis sparsity via $L_1$ constraints (Chung et al., 2023); total variation on output images (Zhang et al., 2022).
• Optimization: Adam-based joint optimization over all point and appearance parameters, camera, and illumination, with pruning and adaptive upsampling. The entire pipeline is run end-to-end, or in alternating schedules for distinct submodules.

PBDR enables not only high reconstruction accuracy—a function of both expressive point attributes and efficient differentiable rendering—but also orders-of-magnitude improvement in training time versus volumetric radiance fields:

• DPIR achieves $2$-hour training for DiLiGenT-MV (PSNR $43.21$ dB), compared to $22$ hours for PS-NeRF (PSNR $35.19$ dB), with $8\times$ less memory (Chung et al., 2023).
• PBDR speeds up NeRF-like view synthesis by $100$–$300\times$ in training and $750\times$ at inference, at modest quality sacrifice (Zhang et al., 2022).

5. Scaling and Computational Infrastructure

While early PBDR systems targeted small to moderate point clouds ($<10^5$–$10^6$ points), efficient large-scale implementations up to hundreds of millions of points have been enabled by distributed training.

The Gaian system (Zhao et al., 23 Dec 2025) introduces a general distributed training architecture with:

• View/point partitioning using bipartite graph partitioning (e.g., METIS) to maximize data locality,
• An API of three primitives: culling, splatting, rendering,
• Two-stage locality optimization (offline partitioning, online linear-sum assignment) to co-locate views and points, reducing communication by up to $91\%$,
• Full compatibility with 3D/2D Gaussian and convex splatting, and dynamic (4D) point-based scenes.

This distributed infrastructure has enabled state-of-the-art throughput (up to $3.7\times$ baseline), real-scale training of $500$ million point scenes with $29.5$ billion parameters, and PSNRs exceeding $26.75$ in aerial reconstruction benchmarks.

6. Applications Across Computer Vision and Graphics

PBDR has demonstrated significant impact in several domains:

• Direct Inverse Rendering: Shape and SVBRDF recovery under controlled or environment lighting (Chung et al., 2023).
• Novel-View Synthesis: View-dependent radiance field learning, with real-time rendering and strong temporal coherence for static and dynamic scenes (Zhang et al., 2022, Rückert et al., 2021).
• Geometry Processing: Self-supervised upsampling and denoising of sparse point clouds, with differentiable render loss enforcing consistent visual appearance and boundary alignment (Zhou et al., 2024, Zhao et al., 2021).
• Multi-view Stylization and Harmonization: Per-view optimization of color and depth, with neural rendering informed by bi-directional EWA splatting (Kopanas et al., 2021).
• Scalable Scene Reconstruction: Efficient, locality-optimized distributed training pipelines for city-scale 3D mapping and photogrammetry (Zhao et al., 23 Dec 2025, Zhang et al., 2022).

7. Limitations and Future Directions

Challenges in PBDR include:

• Handling of extreme sparsity or holes, which can challenge neural shading completion in models such as ADOP (Rückert et al., 2021).
• Sensitivity of approximate gradients, particularly with hard culling steps or front-to-back sorting, requiring regularization and careful selection of parameter learning rates (Rückert et al., 2021, Chung et al., 2023).
• Partitioning and load balancing in distributed systems can degrade with irregular scene topology (Zhao et al., 23 Dec 2025).
• Extensions beyond isotropic point splatting (e.g., non-Gaussian, non-convex atoms) and dynamic scenes with nonrigid motion.

Ongoing work focuses on more expressive splatting primitives, hierarchical spatial/temporal partitioning, hybrid mesh–point systems, and hardware/software co-design for splatting acceleration (Zhao et al., 23 Dec 2025). A plausible implication is that PBDR's differentiable, scalable structure will drive joint learning of geometry and appearance at previously unattainable scene scales.