Fidelity-Aware Projection Modules

• Fidelity-Aware Projection Modules are specialized components that compress and adapt high-dimensional features into lower-dimensional, semantically rich representations.
• They employ dual low-rank projections, FiLM-style modulation, and depthwise convolutions to retain crucial spatial and semantic details.
• Variants like Split-FAPM in SG-RIFE demonstrate improved task performance by preserving feature fidelity during both compression and post-warp refinement.

Fidelity-Aware Projection Modules (FAPM) and their variants are architectural modules designed to compress, adapt, and re-project high-dimensional foundation model features into representations of manageable dimensionality while explicitly preserving semantic and structural fidelity. FAPMs are primarily employed to enable the effective transfer of dense vision transformer features—such as those from DINOv3—to specialized downstream tasks where naïve projection leads to a substantial loss of discriminative detail. These modules have been realized in distinct forms, notably the FAPM used in Dino U-Net for medical image segmentation (Gao et al., 28 Aug 2025), and the Split-FAPM developed in SG-RIFE for semantic-guided real-time video frame interpolation (Wong et al., 20 Dec 2025), each with task-specific design principles but a common emphasis on feature fidelity during compression.

1. Architectural Roles and Data Flow Position

In Dino U-Net, the FAPM is positioned between the adapter-augmented DINOv3 encoder and the U-Net decoder, acting as a bridge that distills high-dimensional, multi-scale backbone features before their integration into the decoder’s skip connections. Its fundamental objectives are to (1) compress high-rank feature maps to lower-dimensional embeddings suitable for U-Net decoding, (2) preserve intricate semantic and spatial detail, and (3) retain the contextual cues critical for fine-grained tasks such as boundary delineation. Direct 1×1 convolution projections are empirically inadequate, as they collapse the nuanced information encapsulated in foundation model representations (Gao et al., 28 Aug 2025).

SG-RIFE’s Split-FAPM operates as a two-stage adapter, split into (i) a pre-warp compressor and (ii) a post-warp refiner that surrounds a flow-based spatial warping operation. Here, Split-FAPM serves to compress and recalibrate DINOv3 semantics for injection into the RIFE backbone alongside standard motion features, ensuring that fragile semantic cues are retained in the highly dynamic context of video frame interpolation (Wong et al., 20 Dec 2025).

2. Mathematical Operations and Formulation

The FAPM architecture in Dino U-Net comprises the following main mathematical components, formalized for scale $i$ with adapter output $$C'_i \in \mathbb{R}^{B \times D \times H_i \times W_i}$$:

• Orthogonal Decomposition: Dual low-rank projections yield context and specific embeddings,

$$Z_{ctx,i} = W_{ctx}^\top C'_i, \quad Z_{sp,i} = W_{sp,i}^\top C'_i$$

with $W_{ctx} \in \mathbb{R}^{D \times R}$ (shared across scales), $W_{sp,i} \in \mathbb{R}^{D \times R}$ (scale-specific), and $R$ the low-rank dimension.

• Context-Gated Modulation: A small generator $\mathcal{G}_i$ produces per-pixel affine scaling and shifting,

$$(\gamma_i, \beta_i) = \mathcal{G}_i(Z_{ctx,i}), \quad Z_{mod,i} = \gamma_i \odot Z_{sp,i} + \beta_i$$

where $\odot$ denotes element-wise multiplication.

• Refinement and SE Re-projection:

$$Y_i = W_{r,i}^\top Z_{mod,i}, \quad Y'_i = C_{dwsep}(Y_i), \quad s_i = \sigma(\mathrm{MLP}(\mathrm{GAP}(Y'_i)))$$

$$\hat{Y}_i = Y'_i \odot s_i, \quad S_i = \hat{Y}_i + \mathcal{P}_i(Z_{mod,i})$$

where $C_{dwsep}$ is a depthwise-separable convolution, $\mathrm{GAP}$ denotes global average pooling, and $\sigma$ is a sigmoid activation as part of the squeeze-and-excitation operation.

The Split-FAPM of SG-RIFE is defined by:

• Pre-Warp Compressor: For each DINO feature map $$D_{raw} \in \mathbb{R}^{B \times 384 \times H \times W}$$,

$$F_{feat} = W_{feat} * D_{raw}, \quad [\gamma, \beta] = W_{mod} * D_{raw}$$

$F_{compressed} = \gamma \odot F_{feat} + \beta$

where all convolutions are $1 \times 1$, compressing 384 to 256 channels and modulating via FiLM.

• Post-Warp Refiner: After warping $F_{compressed}$ using optical flow, depthwise-separable convolution, squeeze-and-excitation, and a 1×1 projection are used to align channels with those required by the downstream fusion backbone.

3. Computational Pipeline: Stage-by-Stage Analysis

A stage-wise breakdown of the FAPM in Dino U-Net is as follows:

1. Input: Adapter feature map with shape $B \times D \times H_i \times W_i$.
2. Dual Projection: Obtain $Z_{ctx,i}$ (context, shared) and $Z_{sp,i}$ (specific, per scale), both with reduced rank $R$.
3. Context Generators: Apply per-scale generator $\mathcal{G}_i$ to produce channel-wise scaling ($\gamma_i$) and shifting ($\beta_i$).
4. Modulation: Compute $Z_{mod,i}$ by applying affine FiLM-based modulation to the specific branch.
5. Re-projection: Project $Z_{mod,i}$ to precise decoder channel count using $W_{r,i}$.
6. Spatial Refinement: Process via depthwise-separable convolution.
7. Channel Recalibration: Use squeeze-and-excitation to adaptively reweight channels.
8. Residual Integration: Add shortcut from $Z_{mod,i}$ (via 1×1 conv or identity) for stable optimization.
9. Decoder Interface: Outputs for all scales serve as skip connections for the U-Net decoder.

The Split-FAPM in SG-RIFE comprises:

Stage	Pre-Warp Compressor	Post-Warp Refiner
Operation	1×1 conv + FiLM modulation	dw-separable conv + SE
Input dim	$B \times 384 \times H \times W$	$B \times 256 \times H \times W$ (per scale)
Output dim	$B \times 256 \times H \times W$ (to warp)	$B \times C' \times H \times W$ (FusionNet)

This modular design allows computationally efficient interoperability with downstream flow fields, followed by immediate artifact repair and channel alignment for semantic fusion.

4. Parameterization, Hyperparameters, and Design Rationale

Dino U-Net’s FAPM selects low-rank $R=256$ across all scales to balance compression overhead and information preservation. Shared $W_{ctx}$ encourages cross-scale consistency, while per-scale $W_{sp,i}$ enables adaptation to varying spatial resolutions. Generators $\mathcal{G}_i$ are lightweight (two linear layers), and depthwise-separable convolutions permit spatial context at negligible parameter cost. Squeeze-and-excitation blocks enable the model to adaptively enhance discriminative channels, and residual connections counteract possible compression-induced information loss. Notably, only the adapter, FAPM, and decoder are trainable—DINOv3 remains frozen—supporting memory savings and regularization (Gao et al., 28 Aug 2025).

For Split-FAPM in SG-RIFE, the module’s pre-warp compressor contains two parallel 1×1 convolutional branches: feature (384→256 channels, ~98k params) and modulation (384→512 channels for [γ,β], ~197k params), totaling ≈295k parameters. The post-warp refiner (dw-separable conv, SE block, GELU, residual, 1×1 projection) adds ≈300k parameters per scale. The entire Split-FAPM contributes ≈0.8M trainable parameters within the 5.6M parameters in all adapters/fusion heads. Design choices such as FiLM modulation, exclusion of BatchNorm, and task-specific channel alignment all serve the aim of preserving crucial semantic structure despite aggressive compression (Wong et al., 20 Dec 2025).

5. Empirical Results and Ablations

Dino U-Net (FAPM): Replacing the FAPM with plain 1×1 convolutions results in a drop of 0.56–0.79 in Dice score across scales and a degradation of 0.09–1.75 mm in HD95, directly indicating inferior boundary delineation. Furthermore, the parameter count for the largest (7B) DINOv3 backbone actually increases when using naïve projections as opposed to the shared low-rank FAPM. These results empirically validate the necessity of fidelity-aware, contextually modulated compression for effective segmentation (Gao et al., 28 Aug 2025).

SG-RIFE (Split-FAPM): Although ablations isolating Split-FAPM are not reported, its integration results in substantial improvements for semantic injection: FID decreases by 5.4 and 11.2 points on the HARD and EXTREME splits of SNU-FILM, respectively, compared to baseline RIFE. SG-RIFE achieves performance on par with or outrunning diffusion-based VFI models while maintaining near real-time throughput (0.05 s/frame), and does so while training only 16% of inference parameters. This confirms that lightweight, fidelity-aware adapters can capture most of the perceptual benefit from semantics (Wong et al., 20 Dec 2025).

6. Mechanisms for Fidelity Preservation

Both FAPM in Dino U-Net and Split-FAPM in SG-RIFE deliberately avoid simplistic dimension reduction (plain 1×1 conv), instead using (a) parallel decompositions to separate context from detail, (b) affine or FiLM-style modulations that recalibrate channel activations before or after spatial transformations, and (c) immediate post-projection refinement using SE blocks and residual connections. The use of spatially aware processing (depthwise-separable convolution) facilitates the retention or restoration of fine boundary and texture details crucial for downstream discriminative tasks. The explicit removal of BatchNorm and reliance on activation strategies such as GELU in Split-FAPM is justified by the preservation of fine-grained and motion-sensitive information.

7. Integration Patterns and Generalization

FAPMs and their variants fit naturally between frozen foundation models and lightweight task-specific heads, enabling parameter-efficient adaptation and transfer of semantically rich features. In Dino U-Net, the FAPM forms a critical bridge for high-fidelity semantic transfer in medical image segmentation where fine structure is essential. In SG-RIFE, Split-FAPM’s two-phase adaptation first compresses and modulates raw DINOv3 features prior to spatial warping, then recalibrates after flow alignment, optimizing for artifact removal and scale-matched injection into the fusion backbone. The core design pattern is general across vision tasks requiring large-scale semantic transfer and could plausibly extend to other architectures seeking the interface between foundation backbone features and highly specialized predictors.

• "Dino U-Net: Exploiting High-Fidelity Dense Features from Foundation Models for Medical Image Segmentation" (Gao et al., 28 Aug 2025)
• "SG-RIFE: Semantic-Guided Real-Time Intermediate Flow Estimation with Diffusion-Competitive Perceptual Quality" (Wong et al., 20 Dec 2025)