Focal Guidance in Video Diffusion and Photography

• Focal Guidance (FG) is a methodology that enables precise, context-sensitive control in generative video diffusion and computational photography by addressing condition isolation and leveraging semantic anchors and depth cues.
• FG integrates fine-grained semantic guidance and an attention cache in video diffusion models to restore prompt adherence and mitigate semantic drift, improving model controllability.
• In computational photography, FG uses user-defined focal plane cues and depth segmentation to achieve customizable bokeh effects, with measurable improvements in PSNR, SSIM, and overall image quality.

Focal Guidance (FG) encompasses a set of methodologies designed to provide precise, context-sensitive control over generative and rendering processes by leveraging either semantic anchors in transformer-based video diffusion models or depth/focal plane cues in computational photography. FG has been recently formulated in two distinct but conceptually analogous domains: enhancing controllability and prompt adherence in image-to-video (I2V) diffusion models (Yin et al., 12 Jan 2026), and delivering customizable bokeh effects in computational photography through guided focal plane selection (Chen et al., 2024). Each formulation addresses distinct challenges in model alignment, guidance, and user-directed content generation.

1. Conceptual Overview and Definitions

In video diffusion modeling, Focal Guidance refers to an architectural intervention that selectively restores text-conditioned control in intermediate transformer layers—termed “Semantic-Weak Layers”—which have lost or attenuated semantic responsiveness due to Condition Isolation (Yin et al., 12 Jan 2026). FG injects explicit text-region anchors and attention footprints, directly counteracting the model’s drift toward generic, visually prior-driven outputs.

In computational photography, Focal Guidance designates a mechanism in which user-specified focal plane and aperture cues are supplied to the bokeh rendering network, ensuring correct preservation of scene sharpness within the desired depth range and plausible defocus elsewhere (Chen et al., 2024). FG here includes segmentation, encoding, and architectural fusion of both physical lens properties and estimated depth cues.

2. Focal Guidance in Video Diffusion Models

2.1 Condition Isolation and Semantic-Weak Layer Phenomenon

DiT-based I2V models are typically conditioned on a triplet of input modalities: (i) a VAE-encoded reference image ($z_{ref}$) capturing high-frequency details, (ii) mid-level image features ($c_{img}$), and (iii) text embeddings ($c_{text}$) encoding semantic instructions. These are injected separately, a pattern identified as Condition Isolation. As a result, many intermediate layers (e.g., layers 11–26 in Wan2.1-I2V) lose sensitivity to textual guidance, a lapse quantified via Moran’s I and standard deviation collapse in text-visual similarity maps—leading to semantic drift and insufficient prompt adherence (Yin et al., 12 Jan 2026).

2.2 FG Mechanisms: Fine-grained Semantic Guidance (FSG) and Attention Cache (AC)

FG operates via two complementary modules:

• Fine-grained Semantic Guidance (FSG):
  • Uses CLIP-style token similarity to extract keywords ($t_k$) from the text prompt, localizes these semantically in the image encoder’s output ($v_n$), and computes anchors $v_{anchor,k}$ via weighted averaging:

  $v_{anchor,k} = \sum_n S_{k,n} \cdot v_n$ - Projects $t_k$ and the constructed anchor into the DiT cross-attention value space, fusing spatial context into each text token. Both the cross-attention stream and the U-Net latent ($z_{ref}$) are enriched by the visual anchor:

  $$V_k^{text} \leftarrow V_k^{text} + \lambda_{text} V_k^{vis}$$

  $$z_{ref}^{(u,v)} \leftarrow z_{ref}^{(u,v)} + \lambda_{lat} w_k^{(u,v)} V_k^{vis}$$

• Attention Cache (AC):
  • Computes per-layer attention maps over keywords and latents using cosine similarity, then aggregates attention maps from semantically strong layers into a spatial cache:

  $$\mathcal{A}_{cache}^t = \sum_{\ell=1}^L \alpha_\ell \mathcal{A}_\ell^t$$ - In semantic-weak layers, injects the cache at positions exceeding a threshold:

  $$z_{lw}^t \leftarrow z_{lw}^t + \lambda_{cache} \mathbb{I}\{\mathcal{A}_{cache,k}^t>\tau_{cache}\} \mathcal{A}_{cache,k}^t V_k^{vis}$$

2.3 Slotting and Training Protocol

FG modules are inserted solely into Semantic-Weak Layers, with cross-attention weights fine-tuned on annotated I2V examples at low learning rates ($\sim$1e-5–1e-4), minimizing changes to the rest of the model. FSG anchors are injected prior to multi-head attention, AC is integrated post self-attention and before the residual update.

3. Focal Guidance in Computational Photography

3.1 Focal Plane Extraction and Encoding

Focal Plane Guidance operates by receiving a binary mask over the depth map ($d(x)$) denoting the focal plane, and a scalar aperture setting ($a$). The system segments depth into three bands ($\{B_1,B_2,B_3\}$) via Otsu-style thresholding, with the focal plane band selected by the user:

$$F_p(x) = \begin{cases} 1 & d(x) \in \text{chosen-band} \ 0 & \text{otherwise} \end{cases}$$

The aperture $a$ is encoded via sinusoidal positional embedding into a vector $\ellens \in \mathbb{R}^d$:

$\ellens[k] = \begin{cases} \sin(a / 10000^{2k/d}), & k \text{ even} \ \cos(a / 10000^{2k/d}), & k \text{ odd} \end{cases}$

3.2 Integration in the Network Architecture

FG cues are integrated at multiple scales within a variable aperture bokeh model (VABM), which utilizes a U-shaped multi-scale Mamba backbone. Information is fused by two blocks:

• Multiple Information Fusion Block (MIFB): Combines all-in-focus RGB, depth map, and focal map, employing per-channel attention to accentuate either global depth or focal-plane cues.
• Lens-Fusion Mamba Block (LFMB): Applies spatial mixing via Mamba SSM and infuses the aperture embedding into feature channels.

The overall pseudocode for the decoder scale is:

for each scale s in decoder: L_r_s = upsample and concat previous features Fused = MIFB(L_r_s, I_d_s, F_p_s) X = Conv(Fused) for i in 1..N_LFMB: X = LFMB(X, lens_embedding(a)) next_features = skip_connection(X)

No explicit per-pixel convolution with a circle of confusion is performed; the network implicitly reproduces correct defocus via learned fusion.

4. Empirical Validation and Benchmarking

FG in video diffusion models was evaluated by constructing an I2V benchmark measuring controllability along three axes—Dynamic Attributes, Human Motion, Human Interaction—plus Subject and Background Consistency. FG delivered +3.97% improvement in total score for Wan2.1-I2V (0.7250), and +7.44% for HunyuanVideo-I2V (0.5571), with concentrated gains in controllability dimensions; background/subject scores remained stable (Yin et al., 12 Jan 2026).

FG in bokeh rendering was validated on EBB! and VABD datasets. Ablation studies demonstrated that excluding depth, focal plane, or fusion blocks led to statistically significant drops in PSNR (up to 0.7 dB), with full FG retaining the highest PSNR, SSIM, and lowest LPIPS (Chen et al., 2024). On EBB!, VABM with FG reached a PSNR of 24.83 and SSIM of 0.8815 with only 4.4M parameters and 9.9G FLOPs.

5. User Interaction and Practical Control

FG methodologies enable direct user intervention in both domains:

• In I2V, text prompt keywords are selected by automatic similarity maps and linked to spatial anchors for semantic control.
• In bokeh rendering, users interactively indicate focal regions (via a mask or click), choose from candidate Otsu depth bands, and select aperture via a slider; the system encodes these choices and infuses them into the network at all scales.

Outputs exhibit precise subject sharpness where indicated and physically consistent transition to defocused regions, with bokeh strength adapting as a function of user-guided aperture.

6. Domain Significance and Future Directions

Focal Guidance serves as a unifying principle for restoring model controllability in settings where conditional signals dissipate or are isolated by architectural design. In transformer-based video generation, FG corrects for prompt-adherence deficits without retraining the core model. In computational photography, FG enables physically plausible focus and bokeh manipulation with minimal parameter and computational cost.

This suggests FG may generalize to other multimodal or generative architectures suffering from analogous “condition isolation” phenomena or lacking explicit region-signal binding. A plausible implication is that FG mechanisms—semantic anchoring and response caching—could be adapted for prompt-controlled image editing, multimodal style transfer, or attention-reinforced rendering in diverse conditional generative frameworks.