Interactive Latent Diffusion for Point Clouds

• The paper introduces a hierarchical latent diffusion framework that disentangles global shape and local detail, achieving state-of-the-art metrics on ShapeNet.
• It employs dual denoising processes and structured score networks to support interactive editing through techniques like shape interpolation, voxel-guided synthesis, and CLIP conditioning.
• The models significantly reduce optimization time for user-driven 3D content creation while enhancing fidelity and control in point cloud manipulation.

Interactive point cloud latent diffusion models constitute a class of generative and editing frameworks that leverage denoising diffusion processes in latent spaces for high-fidelity synthesis, manipulation, and controlled editing of 3D point clouds. These models exploit the expressiveness and flexibility of hierarchical latent variable architectures and the controllability enabled by structured score-matching networks or U-Net bottleneck manipulation. Recent advances, notably LION for 3D shape generation (Zeng et al., 2022) and DragNoise for point-based editing (Liu et al., 2024), demonstrate both unconditional shape synthesis and interactive manipulation in the context of latent diffusion models.

1. Hierarchical Latent Spaces and Variational Autoencoding

LION implements a hierarchical variational autoencoder (VAE) with two levels of latent variables for representing 3D point clouds (Zeng et al., 2022). The global shape latent ($z_g$, often denoted $z_0$) is a $D_z$-dimensional vector (typically $D_z \approx 128$), encoding coarse object geometry. The point-structured latent ($Z_p$, denoted $h_0$) is an $N \times (3 + D_h)$ tensor, where $N$ is the number of points (e.g., 2048), and each point is augmented by a small feature vector ($D_h = 1$). This structure allows explicit disentanglement: $z_g$ governs global semantics, while $Z_p$ modulates fine local detail.

Encoders and decoders are based on Point-Voxel CNNs (PVCNNs), integrating point-wise MLPs with sparse 3D convolutions and GroupNorm. The VAE employs a modified ELBO objective: $$\mathcal{L}_{\mathrm{VAE}} = \mathbb{E}_{x \sim p_{\mathrm{data}}} \, \mathbb{E}_{q_\phi(z_g|x),\, q_\phi(Z_p|x,z_g)} \left[ -\log p_\theta(x \mid Z_p, z_g) \right] + \alpha_g \operatorname{KL}\left[q_\phi(z_g \mid x) \Vert \mathcal{N}(0, I)\right] + \alpha_p \operatorname{KL}\left[q_\phi(Z_p \mid x, z_g) \Vert \mathcal{N}(0, I)\right]$$ with annealed weights $\alpha_g, \alpha_p$.

2. Latent Diffusion Processes and Score Network Training

In LION, two discrete forward noising processes (DDPM-style) are defined—one for $z_g$ and one for $Z_p$: $$z_t = \sqrt{1-\beta_t}\, z_{t-1} + \sqrt{\beta_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

$$h_t = \sqrt{1-\beta_t}\, h_{t-1} + \sqrt{\beta_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

Here, $\beta_t$ schedules noise addition over $T$ steps.

After VAE pretraining, the encoder/decoder weights are frozen, and two score networks are trained: $E_g$ for $z_t$ and $E_p$ for $h_t$, with $E_p$ conditioned explicitly on the global latent $z_0$. Training is done with MSE losses between the true and predicted noise. This hierarchical setup ensures local denoising respects global structure.

3. Interactive Capabilities and User-Driven Editing

Interactive manipulation is facilitated via several mechanisms:

• Shape interpolation (“geodesic” morphing): Two shapes $x^A, x^B$ are encoded; spherical interpolation is performed in the Gaussian-prior latent spaces ($z_T^A, h_T^A$ to $z_T^B, h_T^B$), then inverse-mapped for smooth semantic morphing.
• Voxel/noise-conditioned synthesis: The VAE encoders are fine-tuned on rough voxelizations or noisy clouds, permitting tasks like denoising or voxel-guided refinement with the same ELBO objective; forward and reverse diffusion allow interactive “diffuse-denoise” editing for balancing fidelity and diversity.
• CLIP image/text-conditioning: By projecting CLIP embeddings and appending them to conditioning vectors for both score networks, latent diffusion can be steered by 2D images or text for zero-shot 3D reconstruction or text-driven synthesis.
• User-driven latent traversal: Direct movement in $z_g$ alters global semantics (e.g., object proportions), whereas modifying $Z_p$ targets local features.

DragNoise extends interactive point-based manipulation by directly editing the U-Net bottleneck feature at a selected denoising timestep ($t^\star$). The bottleneck $s_{t^\star}$ is optimized via a semantic alignment loss, propagating intended changes through subsequent denoising steps, yielding precise and efficient point-based editing (Liu et al., 2024).

4. Architecture: U-Net Bottleneck Semantics and Diffusion Edit Propagation

In DragNoise, the U-Net encodes the input into multiscale features via encoder blocks, concentrates semantic information in a central bottleneck, and reconstructs via decoder blocks. Empirically, the bottleneck features ($s_t \in \mathbb{R}^{C \times H \times W}$) at $t^\star \sim 30-40$ are semantically rich and remain stable in subsequent steps.

Point-based editing is performed as follows:

• DDIM inversion maps a point cloud to latent noise steps $\{z_T, \ldots, z_{t^\star}\}$.
• Users provide anchor-target point pairs $(a_i, b_i)$; an alignment loss is used to optimize the bottleneck to semantically “drag” local features.
• Edited bottleneck $\hat{s}_{t^\star}$ is inserted into all U-Net passes for $t > t_{\rm refine}$, propagating semantic changes.
• Denoising proceeds to reconstruct the modified point cloud. This approach avoids optimization through multiple layers and steps, mitigating vanishing gradients and instability found in alternatives like DragDiffusion.

5. Empirical Performance and Comparative Evaluation

LION achieves state-of-the-art metrics on ShapeNet classes. For example, 1-NNA/CD and 1-NNA/EMD (lower is better) for airplanes are 67.4% and 61.2%, substantially improving over previous bests (~73.8% and 64.8%). Chairs and cars similarly show notable improvements. Joint evaluation across 13 ShapeNet classes demonstrates robust generalization, outperforming GAN, flow, and other diffusion baselines in distributional metrics (Zeng et al., 2022).

Interactive schemes such as DragNoise reduce optimization time for user edits by over 50% compared to DragDiffusion (∼10 s/image vs. ∼22 s on similar hardware) and achieve superior control and semantic retention, as measured by mean drag distance and image fidelity (Liu et al., 2024). Voxel-guided or noise-conditioned tasks see a tradeoff between adherence (IoU > 0.8) and generative quality (1-NNA ≈ 50–55%).

6. Implementation and Practical Considerations

Key architectural and operational choices influencing performance and interactivity include:

• Use of PVCNN backbones in LION for high-throughput encoding/decoding.
• For LION, unconditional synthesis with 1000-step DDPM (N=2048) takes ≈27s on a V100. With DDIM sampling and reduced steps (e.g., 25), synthesis is ≲1s.
• In DragNoise, one-shot bottleneck optimization (Adam, lr=0.01, maximum 80 iterations) followed by bottleneck substitution during denoising ensures rapid, stable user-driven point manipulation. Pre-editing fine-tuning on a LoRA adapter further boosts reconstruction fidelity.
• Open-source implementations and pretrained configurations are available for reproduction and extension.

7. Significance and Research Trajectory

Interactive point cloud latent diffusion models marry generative quality with flexible, fine-grained user control for 3D content creation and editing. The hierarchical decomposition in LION (Zeng et al., 2022) enables disentangled manipulation at both global and local scales, while DragNoise (Liu et al., 2024) demonstrates robust, efficient point-based editing by leveraging U-Net semantic bottleneck structures. These approaches underpin emerging workflows in digital content creation, interactive shape design, and multi-modal 3D synthesis, with ongoing research addressing sample efficiency, semantic disentanglement, and multi-object generalization.