Homography-Guided Self-Attention

• Homography-guided self-attention is an attention mechanism that uses geometric priors from homography matrices to accurately align features across time and views.
• It applies projective geometry for tasks like road segmentation and multiview pedestrian detection, reducing computational cost and enhancing occlusion recovery.
• Empirical results show that HGSA improves metrics (e.g., mIoU and MODA) while lowering parameters and compute overhead compared to traditional global attention methods.

Homography-guided self-attention refers to a class of attention mechanisms in which geometric priors—specifically, projective mappings (homographies) between images or video frames—define how and where neural features are pooled, fused, or selectively attended to. These mechanisms exploit the known spatial correspondences induced by scene geometry or camera calibration, dramatically reducing computational complexity and enhancing the effectiveness of temporal and multiview feature aggregation. Homography-guided self-attention (HGSA) can be instantiated in both temporal fusion for video (notably for road scene segmentation) and in multiview settings (such as pedestrian detection in calibrated camera arrays), where projective geometry is leveraged to establish precise pixel- or region-wise alignments for attention computation (Wang et al., 2024, Hwang et al., 2022).

1. Theoretical Foundations: Homographies in Projective Geometry

Homography matrices arise in scenarios where points in one image (or time/frame) correspond via a plane to points in another. Under the pinhole camera model and assuming points lie on or near a planar surface, a $3 \times 3$ homography $H$ relates the pixel $p_t = (u, v)^T$ in one image to its correspondent $p_i$ in a second image:

$H = K \left( R - \frac{1}{d} t n^T \right)K^{-1}$

where $K$ is the intrinsic calibration matrix, $(R, t)$ is the relative pose, $n$ is the estimated unit normal to the reference plane, and $d$ is the distance from the camera to the plane. This construction enables efficient mapping of entire feature maps or selected points across time or between views at minimal cost—provided accurate plane-normal estimation and extrinsics are available. Homography estimation underpins both temporal fusion for video and inter-view fusion for multiview camera settings (Wang et al., 2024, Hwang et al., 2022).

2. Homography-Guided Self-Attention in Temporal Fusion

In road line and marking segmentation, temporal cues from sequential frames enable occlusion recovery and robust recognition under adverse visual conditions. The HomoFusion module applies HGSA as follows:

• For each sampled "on-road" pixel $p_t$ in the current frame, homographies warp $p_t$ to corresponding positions $p_i$ in adjacent frames.
• Queries are defined as local feature vectors at $p_t$: $q = F_t(p_t) \in \mathbb{R}^c$.
• Keys and values are the features at cross-frame correspondences: $k_i = v_i = F_i(p_i) \in \mathbb{R}^c$.
• Attention weights are computed via normalized dot-product similarity followed by softmax: $a_i = \langle q/\|q\|_2, k_i/\|k_i\|_2 \rangle$, $w_i = \text{softmax}_i(a_i)$.
• The fused feature is $$F_t^{\text{fused}}(p_t) = F_t(p_t) + \sum_i w_i v_i$$.
• This process has $O(nc)$ per-pixel cost (with $n$ frames), compared to the $O(HWc)$ cost of non-homography global attention.

Accurate and efficient extraction of pixel correspondences depends on robust plane normal estimation, significantly reducing degrees of freedom (to pitch and roll). The approach produces near-parameter-free temporal attention and can be applied per location at chosen pyramid levels, integrating seamlessly into lightweight encoders/decoders with minimal added overhead (Wang et al., 2024).

3. Homography-Guided Self-Attention in Multiview Pedestrian Detection

In multiview scenarios, HGSA is integrated through modules such as the Homography Attention Module (HAM), as detailed in Booster-SHOT:

• Feature maps $F^i \in \mathbb{R}^{C\times H\times W}$ from $N$ cameras are stacked.
• Channel Gate: For each homography plane $k$, channel attention weights are obtained via max/avg pooling followed by an MLP and softmax, resulting in per-plane, per-camera selection of the most informative channels. Top-$K$ channels per plane are selected.
• Spatial Gate: For each homography plane, channel-compressed features are further modulated by 2-layer ConvNet-generated spatial attention maps.
• Each attention-enhanced stream is warped—via its homography—to a bird’s-eye-view (BEV) grid using the known projection matrices, and aggregated across cameras and planes.
• The resulting BEV feature cube is processed by a head network for occupancy heatmap inference.

Stacking multiple homographies (e.g., $D = 2, 4, 6$) samples parallel planes, addressing scene height variance typical in pedestrian detection. Ablations demonstrate that both channel and spatial gating contribute to superior multi-view fusion, with diminishing returns for $D > 6$ (Hwang et al., 2022).

4. Efficiency and Performance in Representative Architectures

A comparative overview can be structured as follows (as presented for ApolloScape and ApolloScape-Night segmentation tasks):

Method	Params (M)	GFLOPs	18 mIoU	36 mIoU	FPS
IntRA-KD	65.6	5159.4	42.1	24.6	10.8
SegFormer	13.5	1048.8	52.3	32.1	43.8
CFFM	15.3	1192.6	53.2	32.7	22.7
MMA-Net	57.9	723.2	52.9	31.4	20.6
HomoFusion	1.24	61.2	59.3	35.9	25.4

In multiview detection (Wildtrack, MultiviewX), HAM improves MVDet’s MODA metric from 88.2 to 89.4, and gains in MODA with increased $D$ stabilize beyond $D=6$. Channel selection ($K=16$) further concentrates cues (Hwang et al., 2022, Wang et al., 2024).

5. Advantages Over Classical Attention and Fusion

Homography guidance restricts attention aggregation to only the true geometric correspondences, yielding several critical advantages:

• Substantially reduced computational cost: direct $O(nHWc)$ rather than $O(H^2W^2c)$.
• Parameter efficiency: no explicit learned Q/K/V projections are required; existing features serve as attention operands.
• Improved occlusion handling: temporal and/or multiview cues are fused through precise pixel alignment, enhancing robustness under occlusion, adverse lighting, or missing views.
• Geometric faithfulness: known scene priors ensure stable and physically interpretable fusion, outperforming generic global attention and volumetric convolutional fusion modules (Wang et al., 2024).
• Demonstrated empirical performance: HGSA improves mean IoU by over 6–8 points relative to CFFM at a small fraction of the compute budget and number of parameters; full HGSA gives +9.9 points in 18-class mIoU over no fusion baseline on ApolloScape (Wang et al., 2024).

6. Implementation Contexts and Best Practices

HGSA modules are typically inserted post-encoder, pre-decoder, and may be applied at multiple feature pyramid levels (e.g., $\times4$ and $\times16$ downsampling). Key implementation elements include robust surface normal estimation (e.g., feature-based Levenberg–Marquardt), carefully chosen channel/plane selection in HAM, and meticulous camera calibration and plane congruence. In training, view-consistent data augmentations, strong regularization, and tailored loss functions (mean squared error, focal loss with subpixel regression) are standard (Hwang et al., 2022, Wang et al., 2024).

7. Extensions and Applicability

Beyond road marking segmentation and pedestrian detection, HGSA principles may be extended to any scenario where strong planar or known geometric priors govern correspondence—e.g., water puddle segmentation, or other sensor fusion challenges where cross-view or cross-temporal alignment is induced by projective geometry. A plausible implication is that as geometric calibration and depth estimation improve, HGSA-based modules can provide even greater efficiencies, particularly in domains with well-defined planar structures and repeated occlusion or appearance shifts (Wang et al., 2024).

---

References:

"Homography Guided Temporal Fusion for Road Line and Marking Segmentation" (Wang et al., 2024) "Booster-SHOT: Boosting Stacked Homography Transformations for Multiview Pedestrian Detection with Attention" (Hwang et al., 2022)