IMU Temporal Action Localization

• IMU-TAL is a segment-based approach that detects and temporally localizes human actions using raw inertial sensor data.
• Recent methodologies adapt single-stage TAL architectures and leverage weakly supervised learning to infer precise action boundaries.
• Benchmark evaluations on diverse IMU datasets demonstrate improved mAP and F1 scores, ensuring coherent and efficient activity segmentation.

Inertial Measurement Unit Temporal Action Localization (IMU-TAL) refers to the task of detecting and temporally localizing human actions within continuous streams of IMU sensor data, in contrast to the traditional fixed-window classification paradigm dominating inertial sensor-based Human Activity Recognition (HAR). IMU-TAL models predict not only the class of each activity but also its precise start and end boundaries, enabling coherent segment proposals that can adapt to the inherently variable durations of realistic human actions. Initial work in this domain systematically adapts single-stage Temporal Action Localization (TAL) architectures from computer vision for application to raw or latent IMU signals and articulates a comprehensive benchmarking and evaluation framework. Recent developments in weakly supervised IMU-TAL (“WS-IMU-TAL”) further relax the annotation requirements by learning from only sequence-level multi-hot labels, enabling scalability across diverse datasets while presenting new technical challenges for precise segment localization (Bock et al., 2023, Li et al., 2 Feb 2026).

1. Formal Task Definition and Motivation

The central objective of IMU-TAL is, given a sequence of multivariate inertial signals $X = \{x_t \in \mathbb{R}^C\}_{t=1}^T$, to infer a set of labeled activity segments: $$\hat{\mathcal{Y}} = \{(\hat c_j, \hat s_j, \hat e_j)\}_{j=1}^{\hat N}$$ where each triplet encodes the class $\hat c_j$, segment onset $\hat s_j$, and segment offset $\hat e_j$ in the discretized time domain. This segment-based approach is designed to address the limitations of fixed-window (“clip”) classification, which assigns a single label per window $x_t$ and is unable to robustly capture variable-length, overlapping, or temporally ambiguous actions. Segment-based TAL techniques learn per-timestamp class probabilities $p_t \in \Delta^{K+1}$ and regress distances to anticipated start $d_t^s$ and end $d_t^e$ times; aggregated predictions are post-processed via non-maximum suppression to maximize segmental coherence (Bock et al., 2023).

In weakly supervised settings, only sequence-level multi-hot labels $y \in \{0,1\}^C$—indicating class presence anywhere in the sequence—are provided for training; the model must infer boundaries implicitly (Li et al., 2 Feb 2026).

2. Input Representation, Preprocessing, and Data Protocols

IMU windows of fixed length $W$ and channel count $C$ are vectorized: $$\mathrm{vec}(x_{sw}) = [x_{1,1}, \dots, x_{W,C}]^T \in \mathbb{R}^{W \cdot C}$$ to maintain architectural compatibility with vision-based TAL models. Preprocessing includes channel-wise normalization: $\bar{x}_t = \frac{x_t - \mu}{\sigma}$ with statistical parameters $\mu, \sigma$ estimated over training data, and optional online augmentation such as additive noise or random axis permutations (Bock et al., 2023).

Benchmarking protocols standardize seven public IMU datasets (SBHAR, Opportunity, WetLab, Hang-Time, RWHAR, WEAR, XRFV2), employing leave-one-subject-out cross-validation or predefined splits. In weakly supervised benchmarks, only aggregate sequence labels are visible during training, with boundary-annotated ground-truth used for post-hoc evaluation (Li et al., 2 Feb 2026).

3. Model Architectures and Training Objectives

IMU-TAL frameworks adapt three families of single-stage TAL models:

• ActionFormer: Utilizes 1D convolutions for feature projection, transformer encoder layers with local self-attention and strided downsampling, multi-scale feature pyramids, and scale-shared decoder heads for segment classification and boundary regression.
• TemporalMaxer: Substitutes transformer encoders with max-pooling, retaining the multi-scale and decoder structure.
• TriDet: Applies Scalable-Granularity Perception (SGP) modules for context modeling and a trident head that separates regression into start, end, and center offset branches for precision (Bock et al., 2023).

The overarching loss combines focal classification $\mathcal{L}_{cls}$ and generalized IoU regression $\mathcal{L}_{reg}$: $$\mathcal{L} = \lambda_{cls} \mathcal{L}_{cls} + \lambda_{reg} \mathcal{L}_{reg}$$ with hyperparameters chosen for optimal balance. Center sampling strategies, background (NULL-class) weighting, and non-maximum suppression mitigate over-fragmentation and enforce boundary coherence (Bock et al., 2023).

Weakly supervised methods evaluated in WS-IMUBench span audio-inspired MIL pooling (DCASE, CDur), image-based weakly supervised detection networks (WSDDN, OICR, PCL), and video-based MIL and temporal refinement (CoLA, RSKP). Training in these models aggregates per-slice or per-proposal class scores to match sequence-level labels, employing attention, clustering, and temporal smoothing to guide proposals in the absence of explicit boundary annotations (Li et al., 2 Feb 2026).

4. Evaluation Protocols, Metrics, and Experimental Results

Assessment in IMU-TAL leverages both frame-level and segment-level detection metrics:

• Frame-level:

Precision $P = TP/(TP+FP)$, Recall $R = TP/(TP+FN)$, F1 score $2PR/(P+R)$, macro-averaged across activity classes.

• Segment-level mean Average Precision (mAP):

Computed as mean AP across classes and tIoU thresholds $T = \{0.3, 0.4, 0.5, 0.6, 0.7\}$:

$$mAP = \frac{1}{|T|} \sum_{\tau \in T} \frac{1}{K} \sum_{c=1}^K AP_c@\tau$$

where a segment prediction is a true positive if $\mathrm{tIoU} \geq \tau$ with a matching class.

• NULL-class (background) accuracy:

Proportion of correctly predicted background intervals.

WS-IMUBench introduces additional diagnostic metrics: segment misalignment ratios (deletion, underfill, fragmentation, insertion, overfill, merge) and detailed error analysis (Bock et al., 2023, Li et al., 2 Feb 2026).

Empirical studies report that single-stage TAL models surpass traditional baselines on all six evaluated datasets, with mAP improvements up to 26% and F1 increases of up to 25% (notably on SBHAR). TAL achieves higher background accuracy by decreasing spurious insertions and produces more coherent, less fragmented activity segments. In weakly supervised settings, audio-derived MIL pooling methods are competitive for long-duration, high-dimensional datasets (mAP $\approx$65%), while image-derived WSOD approaches are substantially less effective (mAP $<$20%). Short activities ($<$3 s) and content-agnostic 1D proposals remain persistent failure modes (Bock et al., 2023, Li et al., 2 Feb 2026).

5. Comparative Analysis, Limitations, and Practical Challenges

The introduction of multi-scale feature pyramids is critical for representing both short and long-duration actions. Self-attentive and SGP-based architectures both capture local and global context; simpler pooling (e.g., TemporalMaxer) may remain competitive. Soft-NMS and calibrated thresholding are essential for segment deduplication. Current IMU-TAL models are primarily offline; online and real-time adaptations are a recognized limitation. The granularity imposed by fixed input windows ($\geq$1 s) can result in missed or poorly localized very short activities (Bock et al., 2023).

Weakly supervised settings further highlight the modality dependence of transfer learning: temporal-domain proposals and MIL methods are consistently more stable than proposal-based approaches derived from image/object detection models. Failure to align pseudo-proposals with true IMU change-points hampers performance (Li et al., 2 Feb 2026).

6. Future Directions and Open Research Topics

Several promising directions for IMU-TAL and WS-IMU-TAL have been identified:

• Development of lightweight online architectures suitable for resource-constrained environments and real-time inferencing.
• Fusion of multi-modal sensory data, specifically video and IMU, within unified TAL frameworks.
• Self-supervised pretraining of SGP/Transformer modules on large, unlabeled IMU corpora.
• Dynamic windowing mechanisms to mitigate limitations in recognizing very short actions.
• IMU-specific proposal generation leveraging change-point detection and boundary-aware learning objectives to improve localization in weak supervision.
• Multi-scale models and global temporal reasoning to enhance the handling of both long-duration and rapid actions.
• Construction of standardized, cross-dataset IMU foundation models and pretext tasks such as masked signal reconstruction and change-point prediction (Bock et al., 2023, Li et al., 2 Feb 2026).

7. Impact, Benchmarking Infrastructure, and Community Resources

IMU-TAL has established a rigorous, segment-based evaluation tradition in the inertial HAR community, introducing reproducible benchmarking templates and codebases. The bridging of fully and weakly supervised localization protocols accelerates progress toward scalable, annotation-efficient HAR. Standardized datasets (e.g., Opportunity, SBHAR, WetLab, Hang-Time, RWHAR, WEAR, XRFV2), shared cross-validation schemes, and unified metrics have catalyzed comparative and transparent evaluation (Bock et al., 2023, Li et al., 2 Feb 2026). Publicly released code, model weights, and protocols are available for further research and validation.