Unified Feature Injection (UFI)

• Unified Feature Injection (UFI) is a paradigm that integrates multi-modal features from diverse sources to enhance model generalization across tasks.
• In generative editing, UFI leverages DDIM inversion to inject inverted residuals, queries, and keys, ensuring unified semantic outputs.
• For object detection, UFI fuses CNN and vision foundation model features with dynamic weighting to mitigate source bias and boost performance.

Unified Feature Injection (UFI) refers to a methodological paradigm for integrating feature representations from multiple sources or modalities into generative or discriminative models during training or inference. UFI enables the transfer of semantic context and domain-agnostic information across image, video, 3D scene, and object detection pipelines. The technique has emerged independently in at least two distinct areas: (1) multi-modal diffusion-based generative editing (Kwon et al., 2024), and (2) debiased source-free object detection using vision foundation models (Cai et al., 19 Jan 2026). Despite differing operational settings, both approaches share the core goal of leveraging externally extracted features—such as self-attention maps or foundation model embeddings—for enhanced model generalization and semantic consistency.

1. Background and Motivation

UFI in generative editing is motivated by the need to unify text-driven semantic editing across images, videos, panoramic imagery, and 3D scene representations using a single 2D text-to-image (T2I) diffusion backbone. Prior art either relied on plug-and-play feature injection for images or shared-attention mechanisms for videos, lacking extensibility to other modalities or unified pipelines (Kwon et al., 2024).

In source-free object detection (SFOD), UFI addresses persistent source bias, where self-training without access to source data causes detectors to overfit to source-domain statistics and fail on target domains. By injecting features from large, frozen vision foundation models (VFMs) such as DINOv2, UFI regularizes the backbone to retain broad, domain-agnostic semantics (Cai et al., 19 Jan 2026).

2. Feature Extraction, Alignment, and Sharing

Diffusion Model Editing

In multi-modal T2I diffusion, UFI leverages DDIM inversion to extract per-image, per-layer U-Net features at each reverse diffusion step $t$:

• Inverted residuals: $\hat{r}^{l,t}_i$
• Inverted attention queries: $\hat{q}^{l,t}_i$
• Inverted attention keys: $\hat{k}^{l,t}_i$

For each sequence (video frames, neighboring 3D NeRF views, panorama patches), features are stored and later selectively injected, distinguishing a designated reference path $i=ref$ versus the others $i=1 \ldots N$ (Kwon et al., 2024).

Object Detection

In SFOD, UFI operates by fusing features from two branches:

• CNN backbone (ResNet-50): produces multi-scale feature maps $C_3$, $C_4$, $C_5$ of increasing semantic abstraction.
• Frozen VFM (e.g., DINOv2 ViT): outputs $$F_\mathrm{DINO} \in \mathbb{R}^{h \times w \times d}$$.

A "Simple-Scale Extension" aligns $F_\mathrm{DINO}$ to CNN resolutions via a $1 \times 1$ convolution and bilinear upsampling, followed by additive fusion with learnable/scaled weights (Cai et al., 19 Jan 2026).

3. Injection Mechanisms and Mathematical Formulation

Diffusion Sampling with UFI

Editing is achieved by modifying the noise prediction equation at each reverse diffusion step:

For the reference image: $$\epsilon_\mathrm{ref}^t = \begin{cases} \epsilon_\theta(z_\mathrm{ref}^t, t, c; [\hat{r}^{l,t}_\mathrm{ref}, \hat{q}^{l,t}_\mathrm{ref}, \hat{k}^{l,t}_\mathrm{ref}, v^{l,t}_\mathrm{ref}(r^t_\mathrm{ref})]), & t > t_\text{edit} \ \epsilon_\theta(z_\mathrm{ref}^t, t, c; [r^{l,t}_\mathrm{ref}, q^{l,t}_\mathrm{ref}, k^{l,t}_\mathrm{ref}, v^{l,t}_\mathrm{ref}]), & t_\text{context} < t \le t_\text{edit} \ \epsilon_\theta(z_\mathrm{ref}^t, t, c), & t \le t_\text{context} \end{cases}$$ For non-reference paths, a mixture of reference- and per-path features is injected (see Section 2.2-2.3 for details in (Kwon et al., 2024)), with precise blending of structure (query/instance) and context (residual/key/value).

The DDIM update: $$z_i^{t-1} = \frac{1}{\sqrt{\alpha_t}}\Bigl(z_i^t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}}\epsilon_i^t\Bigr) + \sqrt{1-\alpha_{t-1}}\epsilon_i^t$$

Object Detection Fusion and Weight Scheduling

For each detection branch scale $\ell \in \{3,4,5\}$: $$F_{\mathrm{DINO}}^{\mathrm{Proj}, (\ell)} = \mathrm{Upsample}(\mathrm{Conv1\times 1}^{(\ell)}(F_\mathrm{DINO})) \in \mathbb{R}^{H/2^\ell \times W/2^\ell \times C_\ell}$$ Additive fusion: $$F_{\mathrm{fuse}}^{(\ell)} = C_\ell + w_\ell F_{\mathrm{DINO}}^{\mathrm{Proj}, (\ell)}$$ The scale factor $w_\ell$ is selected via domain-aware adaptive weighting (DAAW), using joint stability metrics on class confidences and bounding box IoUs over a candidate grid, with warm-up ramping to avoid instability (Cai et al., 19 Jan 2026).

4. Semantic Consistency and Regularization

Diffusion Models

Semantic consistency across modalities is realized by the specific choice of which features to inject and share at different timesteps:

• In 3D scenes, reference-view editing is combined with synchronized context transfer, ensuring geometric and style coherence after NeRF retraining.
• In videos, key/value sharing eliminates flicker and propagates edits smoothly.
• In panoramas, overlapping region blending and patch-wise context injection enforce seam-free global edits.

The phase schedule $(t_\text{context}, t_\text{edit})$ allows adaptive control for different semantic editing strengths.

Object Detection

Additive VFM–CNN fusion can drift the CNN feature manifold during adaptation. UFI is therefore constrained by Semantic-aware Feature Regularization (SAFR):

$$\mathcal{L}_\mathrm{reg} = \frac{1}{HWL}\sum_{i,j,l}H^{(i,j)}\lVert F_\mathrm{inv}^{(i,j,l)} - F_\mathrm{DINO}^{(i,j)} \rVert_2^2$$

$H^{(i,j)}$ is a Gaussian heatmap over pseudo-label boxes, focusing regularization on likely foreground. SAFR enforces that CNN features (projected back to the DINO space) remain anchored to VFM semantics, thus reducing source bias and encouraging domain-agnostic representations (Cai et al., 19 Jan 2026).

5. Implementation Schematics

Diffusion Pseudocode

A Python-style outline demonstrates UFI integration at each diffusion iteration, detailing:

• Precomputation of inverted features via DDIM inversion
• Iterative timestep-based injection logic distinguishing reference and non-reference paths
• Dynamic scheduling (see Section 4 in (Kwon et al., 2024) for code and detail)

Detection Forward Pass

Key forward steps per image:

1. Resize inputs for both branches (CNN, VFM)
2. Compute ResNet multi-scale features ($C_3$, $C_4$, $C_5$)
3. Extract $F_\mathrm{DINO}$ from ViT
4. SSE: project, align, and upsample VFM features
5. Additive fusion with current $w_\ell$
6. Input fused maps into transformer detector head(s)
7. During adaptation: UFI for student, pseudo-label guidance from an EMA-updated frozen teacher (Cai et al., 19 Jan 2026)

6. Quantitative Benchmarks and Ablation Analyses

Generative Editing

UFI establishes new state-of-the-art metrics across modalities (Kwon et al., 2024):

Modality	Metric	Baseline (Best)	UFI
3D Scenes	CLIP Dir. Score $\uparrow$	0.1835 (ED-NeRF)	0.2351
	View Consistency $\uparrow$	0.9512 (NeRF-Art)	0.9480
Panoramas	CLIP Text Sim $\uparrow$	0.1812 (CSD)	0.2053
	LPIPS Structure $\uparrow$	0.6232 (CSD)	0.5725
Videos	CLIP Text Sim $\uparrow$	0.2340 (Gen-1)	0.2284
	Frame Consistency $\uparrow$	0.9541 (CSD)	0.9473

Ablations demonstrate that removing any of the injected components ({resnet, query, key, value}) severely degrades outputs.

Object Detection

In SFOD, UFI significantly boosts adaptation performance (Cai et al., 19 Jan 2026):

• Cityscapes$\rightarrow$Foggy Cityscapes: Baseline 39.6 AP; UFI 43.0 AP; with full DSOD (UFI+SAFR+DAAW+MIC) 48.1 AP.
• Cityscapes$\rightarrow$BDD100k: Baseline 36.6 AP; DSOD 39.2 AP.
• SIM10k$\rightarrow$Cityscapes: DSOD 61.4 AP, outperforming DDT (60.6 AP) and DRU (58.7 AP).

DAAW reaches optimal injection weights near $w \approx 0.4$; excessive weighting leads to performance collapse.

7. Applications and Limitations

UFI enables:

• Multi-modal, unified and structure-consistent editing of 3D scenes, videos, and panoramas without retraining or auxiliary models, using only 2D T2I diffusion backbones (Kwon et al., 2024).
• Robust source-free object detection that generalizes across domain shifts by injecting VFM feature regularization, mitigating the lack of source data and controlling for domain bias (Cai et al., 19 Jan 2026).

A limitation in the object detection context is that additive VFM fusion, if unconstrained, can destabilize the CNN feature manifold; thus, mechanisms such as SAFR and DAAW are required to ensure robust adaptation.

Qualitative PCA analyses in detection confirm preservation of both instance (CNN) and semantic (VFM) feature groupings, supporting the claim of orthogonal, information-preserving fusion.

---

In summary, Unified Feature Injection constitutes a lightweight and extensible approach for both semantic-consistent generation across modalities and for domain-agnostic, unbiased object detection. The method is characterized by invert-reinject architectures in generative models and additive, weighted feature fusion in detection models, supported by rigorous regularization and dynamic scheduling. This approach demonstrates substantial performance improvements across generative and discriminative tasks by seamlessly integrating multi-source semantic signals (Kwon et al., 2024, Cai et al., 19 Jan 2026).