Camera Motion-Specific LoRA Modules

• The paper demonstrates how camera motion-specific LoRA modules enable fine-grained, disentangled control by decomposing motion primitives using low-rank adaptations.
• It employs subspace orthogonality, norm consistency regularization, and scaling tokens to stabilize multi-LoRA fusion and maintain independent control over spatial and temporal dynamics.
• The approach facilitates separate manipulation of camera and object motion, offering robust applications in personalized trajectory synthesis and image-to-video transfer.

Camera Motion-Specific LoRA Modules are modular adaptation layers, leveraging the Low-Rank Adaptation (LoRA) paradigm, that are designed to capture, control, and transfer distinct categories of camera motion within video diffusion models. By structurally decomposing motion primitives and ensuring subspace orthogonality, these modules facilitate fine-grained, disentangled control over spatial and temporal dynamics in generative video frameworks. Recent advances, such as LiON-LoRA (Zhang et al., 8 Jul 2025) and CamMimic (Guhan et al., 13 Apr 2025), demonstrate the principles and effectiveness of this methodology for generating and editing video content with explicitly modeled camera trajectories.

1. Architectural Integration and Formulation

The core mechanism underlying camera motion-specific LoRA modules is the insertion of parallel, low-rank, trainable weight updates at pivotal sites within the backbone of a video diffusion model, typically at each linear or self-attention layer of a transformer-based architecture.

For a given set of camera motion primitives (e.g., pan, dolly, orbit), separate LoRAs are injected such that the $i$-th primitive at the $l$-th layer induces an adapted weight:

$W_i^l = W^{\text{base},l} + \lambda_i \Delta W_i^l$

$$\Delta W_i^l = A_i^l B_i^l,\qquad A_i^l \in \mathbb{R}^{k\times r},\; B_i^l \in \mathbb{R}^{r\times d},\; r \ll \min(k, d)$$

where $W^{\text{base},l}$ is the base weight (frozen during adaptation), and $A_i^l$, $B_i^l$ contain the only trainable parameters for each primitive (Zhang et al., 8 Jul 2025). In CamMimic, a related decomposition is applied per attention block, with separate LoRA modules for spatial (per-frame appearance) vs. temporal (cross-frame motion) self-attention (Guhan et al., 13 Apr 2025).

This modularity enables the learning of compact, well-structured subspaces for distinct motion types and allows for independent manipulation at inference.

2. Subspace Orthogonality and Motion Disentanglement

To preserve the disentanglement of motion primitives and prevent interference during fusion, camera motion-specific LoRA modules enforce (or empirically observe) subspace orthogonality in the following ways:

• Empirical Orthogonality: In LiON-LoRA, low-rank updates $\{\Delta W_i^l\}$ corresponding to different primitives are observed to be nearly orthogonal in shallow layers—those responsible for encoding low-frequency camera effects—with average cosine similarity $\approx 0.06\pm0.06$ (Zhang et al., 8 Jul 2025).
• Orthogonality Regularization: An explicit penalty can be applied, such as

$$\mathcal{L}_\mathrm{ortho} = \sum_{l\in\mathcal{L}_\mathrm{shallow}} \sum_{i\neq j} \|(\Delta W_i^l)(\Delta W_j^l)^\top\|_F^2$$

to enforce $\langle \Delta W_i^l, \Delta W_j^l \rangle = 0$ for $i\neq j$.

• Spatial vs. Temporal Decoupling: CamMimic imposes an orthogonality loss between spatial and temporal LoRA weights ($$L_\mathrm{ortho} = \langle W_\mathrm{spatial}, W_\mathrm{temporal} \rangle$$), ensuring that spatial (scene) adaptation does not corrupt the motion (camera trajectory) subspace during image-to-video transfer (Guhan et al., 13 Apr 2025).

Such mechanisms ensure modularity and prevent mode collapse or unwanted entanglement during multi-primitives fusion or scene transfer scenarios.

3. Norm Consistency and Multi-LoRA Fusion

When independently learned LoRA modules are fused (i.e., multiple motion primitives are activated simultaneously), discrepancies in update magnitudes can cause instability and dominance of one primitive. To address this, norm consistency regularization is employed:

$$\Delta\hat W_i^l = \frac{\alpha^l}{\|\Delta W_i^l\|_F} \Delta W_i^l,\qquad \alpha^l = \frac{1}{k} \sum_{i=1}^k \|\Delta W_i^l\|_F$$

This procedure post-normalizes each update so that $\|\Delta\hat W_i^l\|_F \approx \alpha^l$ across all $i$, stabilizing the multi-LoRA fusion process (Zhang et al., 8 Jul 2025). As a result, composite motion can be synthesized without the dominance of any single primitive, supporting robust compositional control.

4. Controllable Scaling Tokens and Self-Attention Modulation

Explicit, fine-grained control over the amplitude of each motion primitive is achieved via learned scaling tokens. In LiON-LoRA, a scaling token $\mathcal{E}_i \in \mathbb{R}^d$ encodes the desired motion strength $S_i \in [s, 1]$ through a Fourier embedding and MLP projection:

$$\gamma(S_i) = \left[ \sin(2^0\pi S_i), \cos(2^0\pi S_i), \dots, \sin(2^{J-1}\pi S_i), \cos(2^{J-1}\pi S_i) \right]$$

$\mathcal{E}_i = \mathrm{MLP}(\gamma(S_i))$

At each transformer block, the hidden sequence $H \in \mathbb{R}^{n\times d}$ is augmented:

$$H'_i = [H;\mathcal{E}_i] \in \mathbb{R}^{(n+1) \times d}$$

A modified self-attention operation is then applied:

$$\operatorname{Attn}_i(H'_i) = \operatorname{softmax}\left( \frac{(H'_i W^Q)(H'_i W^K)^\top}{\sqrt{d}} \right) (H'_i W^V)$$

This design ensures that LoRA updates are responsive to external control and remain orthogonal to other latent subspaces. When multiple controls are used at inference, each receives a separate scaling token and sub-attention pass, with results averaged for the main tokens (Zhang et al., 8 Jul 2025).

5. Separability of Camera and Object Motion

Camera motion-specific LoRA modules are explicitly decoupled from object-motion counterparts through separate training and non-overlapping adapter/token allocations:

• Two disjoint LoRA sets are trained: one for camera-moving (static-object) videos, another for object-motion (static-camera) sequences.
• Each set possesses unique scaling tokens.
• At inference, this yields mutually exclusive control over camera and object dynamics.

Owing to the linearity of $S_i$ in the scaling token embedding, a nearly linear relationship is established between the control parameter and mean camera angle/speed or object flow, with linearity confirmed via Pearson correlation between $S$ and optical-flow magnitude (Zhang et al., 8 Jul 2025).

6. Training Pipelines, Evaluation, and Inference Procedures

The training and application of camera motion-specific LoRA modules follows structured procedures:

• LiON-LoRA Pipeline:
  • For each motion primitive: render 100 scene instances (3DGS+DL3DV), producing 600-frame videos.
  • For each $S \in [s, 1]$, sample 49 frames from first $\lfloor 600S \rfloor$.
  • Fine-tune only $A^l, B^l$ (with $r=256$) and the scaling MLPs for 4k steps (LR $5\times10^{-4}$) on 4 H20 GPUs.
  • Evaluation metrics: Rotation Error ($\mathrm{RotErr}$), Translation Error ($\mathrm{TransErr}$), Absolute Trajectory Error (ATE), FVD, and Pearson $\rho$ for object motion (Zhang et al., 8 Jul 2025).
• CamMimic Two-Stage Strategy:
  • Stage 1: Adapt both spatial and temporal LoRAs on a reference video $V_R$, using diffusion noise-prediction losses regularized by $\delta$ for balancing modalities.
  • Stage 2: With temporal LoRAs frozen, fine-tune spatial LoRAs on image $I_u$ (target scene), using a loss $$L_\mathrm{second-stage}=L_\mathrm{spatial}+\lambda L_\mathrm{ortho}$$.
  • Homography-based refinement: At inference, align latent generations toward homography-warped versions of $I_u$, encouraging spatial-temporal consistency via gradient-based guidance (Guhan et al., 13 Apr 2025).

A summary table compares key design choices:

System	Primitive Decoupling	Norm Consistency	Scaling Token	Orthogonality Loss	Inference Control
LiON-LoRA	Yes	Yes	Yes	Optional	Linear, per-token
CamMimic	Yes (spatial/temporal)	Not explicit	Not direct	Yes	Scene transfer

7. Applications, Metrics, and Broader Impact

Camera motion-specific LoRA modules enable explicit and compositional control of video generation, supporting personalized camera trajectory synthesis, motion transfer across scenes, and disentangled modulation of spatial and temporal content.

• In CamMimic, zero-shot image-to-camera-motion transfer has been demonstrated with Zeroscope v2 as the backbone (Guhan et al., 13 Apr 2025).
• Novel metrics such as CameraScore (homography-based similarity) provide objective assessment of camera motion transfer fidelity.
• State-of-the-art performance on trajectory control accuracy, motion strength adjustment, and generalization with minimal training data is reported for LiON-LoRA (Zhang et al., 8 Jul 2025).

A plausible implication is the broader utility of such LoRA-based modularization for other disentanglement and control tasks within generative video, animation, and robotics domains, provided orthogonality and norm consistency can be ensured. The explicit architectural separation and fine-grained control strategies established in these systems lay groundwork for subsequent advances in interpretable and modular generative modeling.