Temporal Proposal Generation in Video Analysis

• Temporal proposal generation is the process of hypothesizing candidate temporal intervals in untrimmed videos, balancing high recall and precise boundary localization.
• It employs methods such as boundary pairing, anchor-based prediction, and dense confidence mapping to efficiently reduce search spaces and enhance detection performance.
• Recent advances integrate transformer and graph-based architectures for context aggregation and enhanced proposal ranking, leading to improved action localization benchmarks.

Temporal proposal generation is the process of hypothesizing candidate temporal intervals within untrimmed videos that are likely to contain action instances. These candidate intervals—temporal proposals—form the essential input for downstream temporal action detection and localization systems, as they reduce the search space from all possible segments to a tractable yet highly-recall set of intervals. Temporal proposal generation must simultaneously achieve high recall across action instances, precise localization of action boundaries, and robust ranking for downstream retrieval. The field comprises a spectrum of methodologies including boundary-based pairing, anchor-based prediction, dense confidence map sampling, proposal-level relational modeling, and more recently, transformer and graph-based context aggregation frameworks. Advances in temporal proposal generation have driven improvements in action localization benchmarks including ActivityNet-1.3, THUMOS14, and HACS Segments.

1. Core Problem and Evaluation Metrics

Temporal proposal generation operates on feature sequences $X \in \mathbb{R}^{C \times L}$ derived from untrimmed videos, where $C$ is the clip feature dimension and $L$ is the number of clips per video (Chen et al., 2022). The objective is to output a set of candidate intervals $\Psi_p = \{ (t_{s,k}, t_{e,k}, p_k) \}_{k=1}^{K}$, with $(t_{s,k}, t_{e,k})$ denoting start/end indices and $p_k$ a confidence score.

The standard metrics are:

• Average Recall at Average Number of proposals (AR@AN): Defined as the mean recall of ground-truth actions at a fixed average number of predicted proposals per video, for a range of temporal Intersection-over-Union (tIoU) thresholds (Lin et al., 2018, Lin et al., 2019).
• Area Under the AR-AN Curve (AUC): AR plotted across AN, then integrated.
• Mean Average Precision (mAP): Used when proposals are combined with classifiers for full action localization (Chen et al., 2022).

2. Boundary and Confidence Modeling Paradigms

Two dominant paradigms exist for proposal generation: boundary-sensitive pairing and anchor-based (region proposal) scoring.

Boundary-based pairing: This method estimates start and end boundary probabilities at each time index and forms proposals by pairing start/end candidates (Lin et al., 2018, Lin et al., 2019). Key steps include:

• Temporal Evaluation Module (TEM): 1D-CNN predicts $p^s(t)$, $p^e(t)$—boundary probabilities per index.
• Proposal Evaluation Module (PEM): Computes scores for candidate intervals, classically using either learned MLPs over pooled feature representations (Lin et al., 2018) or boundary-matching confidence maps for all $(i,j)$ pairs (Lin et al., 2019).

Anchor-based and dense map approaches: Proposals are predicted using anchor intervals at multiple temporal scales and locations, or by efficiently evaluating all pairs through dense tensors:

• Anchor-based generators: Predefined temporal anchors (center, duration) at each feature location, with regression to refine boundaries (Liu et al., 2018, Gao et al., 2020).
• Boundary-Matching maps: $\mathbb{R}^{D \times T}$ confidence maps where each entry $(i,j)$ scores a proposal of length $i$ starting at $j$ (Lin et al., 2019, Chen et al., 2022).
• Proposal pruning: Top-K scoring intervals are selected, often followed by non-maximum suppression (NMS) or Soft-NMS to limit redundancy (Chen et al., 2022, Lin et al., 2018).

3. Architectural Advances: Context, Relations, and End-to-End Learning

Recent frameworks extend basic pairing/anchor logic with richer temporal and inter-proposal context modeling:

Context-Adaptive and Relational Blocks:

• Self-/cross-attention mechanisms: Exampled by the Context-Adaptive Proposal Module in Faster-TAD (Chen et al., 2022), which applies transformer decoder and cross-attention layers over proposal features and the full video context, capturing long-range dependencies among proposals and between proposals and the temporal feature sequence.
• Proposal Relation Block (PRB): As in BSN++ (Su et al., 2020), encodes inter-proposal dependencies via position- and channel-wise self-attention to improve proposal ranking and reliability.
• Explicit graph reasoning: BC-GNN treats proposal boundaries and content as nodes and edges in a graph neural network, refining both via edge/node updates and message passing to yield state-of-the-art recall (Bai et al., 2020).

Label Assignment and Training:

• Soft and proximity labels: Beyond binary positive/negative assignment, frameworks like Faster-TAD use 2M-category soft labeling, accounting for proximity to ground-truth through measured IoU and distance (Chen et al., 2022). Scale-balanced sampling is also employed to ensure robust training across proposal durations (Su et al., 2020).
• Multi-task objectives: Losses blend classification (boundary, proposal) and regression (offset) terms, with focal or smooth-L1 losses to manage class imbalance and outlier proposals (Chen et al., 2022, Su et al., 2020).

Auxiliary features and proposal augmentation:

• Fake proposals and atomic features: Artificial proposals with offset boundaries are injected to improve regressor robustness (Chen et al., 2022). Auxiliary atomic-action features extracted from complementary networks are fused to increase proposal discriminability under fine-grained motion (Chen et al., 2022).

4. Transformer and Graph Architectures

Advanced context modeling is central to recent state-of-the-art:

• Transformer-based networks: TAPG Transformer (Wang et al., 2021) separates boundary detection and proposal confidence evaluation into two transformer modules (Boundary Transformer, Proposal Transformer) to decouple frame-wise locality from proposal-level dependencies.
• Unified transformer detectors: RTD-Net (Tan et al., 2021) discards anchors, using a set of proposal queries and a relaxed Hungarian matching scheme with an auxiliary completeness prediction head, producing NMS-free proposals with high accuracy.
• Hybrid local-global encoders: ATAG (Chang et al., 2021) utilizes an augmented transformer for global context and an adaptive graph convolutional network for local, position-sensitive context. Fused features drive more accurate proposal boundaries, notably under background clutter.
• Pyramid slot-based local attention: PRSA-Net (Li et al., 2022) leverages a pyramid region-based slot-attention encoder-decoder, focusing only on local neighborhoods across multiple parallel scales, outperforming global similarity computation approaches.

5. Proposal Diversity, Imbalance Mitigation, and Scalability

To counteract intrinsic biases and organization mismatches:

• Scale-balanced and IoU-balanced sampling: BSN++ implements a two-stage re-sampling strategy to equalize the influence of proposal duration, improving recall for both short and long actions (Su et al., 2020).
• Synthetic/fake proposals: As in Faster-TAD, a fixed fraction of proposals per batch are synthesized by offsetting ground-truth boundaries, exposing the network to a range of errors during training (Chen et al., 2022).
• Parallel and distributed training: Scalability on large datasets is supported by frameworks such as MTN with high-performance MPI-based ring communication, achieving near-linear speedup for proposal training on multi-GPU clusters (Wang et al., 2019).

6. Empirical Performance and Benchmarks

Temporal proposal generation methods are primarily evaluated on ActivityNet-1.3, THUMOS14, and HACS Segments, with recall and mAP as principal metrics. Recent results include:

• Faster-TAD: ActivityNet-1.3 mAP 40.01%, HACS 38.39%, SoccerNet Action Spotting 54.09%, outperforming prior single-network detectors (Chen et al., 2022).
• PRSA-Net: On THUMOS14, AR@100 reaching 56.12% and mAP at 0.5 tIoU of 58.7% (UntrimmedNet) and 55.0% (P-GCN), exceeding all prior reported numbers (Li et al., 2022).
• ATAG: On ActivityNet1.3 (val), AUC=68.50%, AR@100=76.75%; THUMOS14 (AR@100)=52.21% (Chang et al., 2021).
• BSN++: ActivityNet-1.3 AUC=68.26%, THUMOS14 AR@100=49.84% (Su et al., 2020).
• RTD-Net: THUMOS14 AR@100=49.32%, no NMS needed (Tan et al., 2021).
• TCANet: ActivityNet-1.3 AUC=68.08%, THUMOS14 AR@100=50.48%, with local-global refinement (Qing et al., 2021). These results indicate a continual increase in proposal recall and precision as context modeling, label assignment, and training protocols evolve.

7. Synthesis and Prospective Directions

Contemporary research in temporal proposal generation emphasizes:

• Multi-level and relational context aggregation via transformers, graph neural networks, and self-/cross-attention structures to improve both boundary precision and proposal discriminability.
• Balanced label assignment and diversity-promoting training to counteract long-tailed duration distributions and boundary misalignment biases.
• Scalability and efficiency, including ring-based distributed training and sampling-efficient architectures, to handle modern video corpora at scale.

Ongoing areas of investigation include end-to-end unified detection/classification models, dynamic fusion of agent-centric and environmental context, and explicit modeling of segmental relationships beyond pairwise boundaries. The trend is towards more holistic, context-aware, and self-supervised pretraining pipeline elements to further push recall and localization precision in complex real-world scenarios (Chen et al., 2022, Su et al., 2020, Wang et al., 2021).