Composite Relative Temporal Encoding

• Composite relative temporal encoding is a method that combines relative timing and multi-dimensional compositeness to represent temporal data robustly.
• It underpins various applications such as 3D pose estimation, time series classification, event-based vision, and video modeling by disentangling dynamic relational dependencies.
• The approach achieves improved model interpretability and efficiency through techniques like feature binding, relative subtraction, and multi-channel fusion.

Composite relative temporal encoding refers to a family of methods that augment temporal data representations with features expressing both relative timing and composite structure across multiple temporal or semantic dimensions. These approaches are motivated by the desire to disentangle or emphasize relational and contingent temporal dependencies, achieving robust invariance to global shifts, enabling fine-grained motion sensitivity, and facilitating associative and compositional retrieval. Widely adopted in time series analysis, 3D human pose estimation, low-latency event-based vision, video recognition, and biologically inspired memory models, composite relative temporal encoding has proven effective both empirically and theoretically in reducing error rates, enhancing robustness, and improving model interpretability.

1. Core Principles of Composite Relative Temporal Encoding

Composite relative temporal encoding unifies two distinct but synergistic concepts:

• Relative encoding: Features are expressed as differences or relationships with respect to a reference, rather than absolute coordinates or indices. This includes, for example, subtraction of a root or central point (as in pose estimation), or pairwise differences across time or space.
• Compositeness: The encoding captures multiple, complementary relational structures—for instance, spatial and temporal, static correlations and dynamic transitions, or feature–role bindings—within a single representation. These are either stacked as multichannel tensors or bound via mathematical composition.

This dual strategy discourages a model from overfitting to global position or timing artifacts and instead promotes learning of stable, meaningful relational invariances present in temporal data (Shan et al., 2021, Wang et al., 2015).

2. Mathematical Formulations Across Modalities

Table: Formulations in Key Domains

Domain / Paper	Relative Component	Composite/Temporal Component
3D Pose Estimation (Shan et al., 2021)	$\mathcal{K}_P$: $\mathbf{k}_t^j-\mathbf{k}_t^0$	$\mathcal{K}_T$: $\mathbf{k}_t^j-\mathbf{k}_{t_c}^j$
Time Series Imaging (Wang et al., 2015)	GAF: pairwise angular cosines	MTF: transition probabilities
Event-Based Vision (Innocenti et al., 2020)	Pixelwise occurrence across slices	Binary-to-decimal fusion over $N$ slices
Video Rel. Pos. Encoding (Hao et al., 2024)	Groupwise relative bias matrices $P_T^g$	Spatio-temporal gating/fused RPE
STDP Composite Memory (Yoon et al., 2021)	Tensor-product binding $f_i\otimes r_{k(i)}$	Phase-offset driving for ordered storage

In all cases, a composite of at least two relative structures—temporal and another (e.g., spatial, semantic, anatomical)—is formed, conveying richer context than either component alone.

3. Domains and Architectures Employing Composite Relative Temporal Encoding

3D Human Pose Estimation augments standard 2D-to-3D lifting networks by incorporating both a global-translation-invariant positional encoding (joint positions relative to pelvis) and a temporal differential encoding (per-joint displacement relative to the window’s central frame). Inputs are grouped anatomically and processed by temporal convolutional networks; feature fusion modules and staged training procedures yield further improvements. Quantitatively, this approach reduces error (mean per-joint position error, MPJPE) by up to 4 mm on benchmarks and notably increases robustness to spatial jitter and subtle local motion (Shan et al., 2021).

Time Series Classification leverages composite image encodings—the Gramian Angular Field (GAF) for static pairwise angular relationships and the Markov Transition Field (MTF) for dynamic state transitions—stacked to create a 2-channel image fed to convolutional networks. The fusion is empirically superior, achieving test error reductions and improved feature diversity, specifically because the quasi-orthogonality of GAF and MTF extracts complementary static and dynamic regularities (Wang et al., 2015).

Event-Based Action Recognition (e.g., DVS cameras) encodes temporal structure by transforming raw event streams into multiple binary slices (one per temporal window), then compresses them into grayscale images using bitwise fusion (binary-to-decimal conversion). This aggregates when-in-time pixel activity in a relative manner, enabling deep models to exploit timing patterns for gesture classification. Ablations show that carefully tuned interval length and bit depth produce the best accuracy (e.g., 99.62% on DVS128) (Innocenti et al., 2020).

Spatio-Temporal Video Modeling (e.g., PosMLP-Video) factorizes MLP blocks into separate gating functions along spatial and temporal axes, employing lightweight, learnable dictionaries of relative position biases. Distinct gate units such as the Temporal Positional Gating Unit (PoTGU) and Spatio-Temporal Gating Unit (PoSTGU) pool contextual information via these biases. Channel-grouping multiplies relative bias capacity, and such composite gating achieves strong performance/efficiency tradeoffs relative to dense attention layers (Hao et al., 2024).

Biologically Inspired Associative Memory (Yoon et al., 2021) implements compositional storage and recall, binding high-dimensional features to role- or position-tags, then inducing relative temporal structure via phase-offset sinusoidal inputs. Spike-timing-dependent plasticity (STDP) sculpts a 2-dimensional oscillatory memory plane; recall is governed by phase-locked oscillations elicited by cues. This enables both exclusive and simultaneous retrieval of overlapping compositional memories, depending on cue specificity (e.g., recalling all sentences sharing a particular subject).

4. Empirical and Theoretical Benefits

Composite relative temporal encoding confers several experimentally verified and theoretically justified advantages:

• Robustness to global shifts and translation artifacts: Subtraction of global components (pelvis in pose; global motion in scenes) directly nullifies sensitivity to positional or temporal drift. For example, incorporating $\mathcal{K}_P$ in pose networks decreased MPJPE variance under global 2D shifts by >15% (Shan et al., 2021).
• Amplification of fine-grained local motion: Temporal-relational representation ensures that even small-magnitude dynamics are explicit; error on lowest-motion pose sequences improved by 1.6 mm over baseline TCNs (Shan et al., 2021).
• Expressive modeling of both static and dynamic temporal dependencies: The combination of GAF and MTF channels captures information unavailable to single-view encodings, achieving higher accuracy and improved invariances in time series classification (Wang et al., 2015).
• Parameter and computational efficiency: Relative positional bias dictionaries (as in PosMLP-Video) enable parameter scaling from $O(T^2)$ to $O(T)$, with similar reductions in FLOPs compared to dense self-attention (Hao et al., 2024).
• Compositional and associative retrieval: Tag-based binding and phase-offset encodings allow for both exclusive and simultaneous memory recall from overlapping representations, controlled by semantic cues (Yoon et al., 2021).
• Minimal overfitting and increased feature diversity: Tiled CNN and topographic ICA pretraining, as in GAF-MTF models, ensure locally orthogonal filters, mitigating overfitting and enhancing generalization (Wang et al., 2015).

5. Implementation Strategies and Architectural Design

Composite relative temporal encoding is typically implemented through a sequence of steps, with variations in architectural detail depending on modality:

1. Preprocessing/Construction of Relational Features:
   • Subtract per-frame or per-sample reference points to obtain positional or temporal differences (e.g., joint-to-pelvis; frame-to-central-frame; bitwise temporal slices).
   • Compute pairwise, Gramian, or Markovian structures (e.g., GAF, MTF).
   • Bind features to semantic or grammatical tags by tensor product.
2. Feature Fusion:
   • Stack, concatenate, or channel-group composite encodings for joint processing.
   • Input multi-channel composites to specialized neural blocks (e.g., TCN, Tiled CNN, MLP with gating).
3. Network Processing:
   • Temporal ConvNets, gated MLPs (PoTGU/PoSTGU), or recurrent modules extract both immediate and distributed temporal relationships.
   • Feature fusion modules enforce consistency and enable joint context modeling across anatomical, spatial, or channel groups.
4. Training and Optimization:
   • Multi-stage optimization may be used to isolate learning in local encoders before full joint/fusion training (e.g., sequential freezing and unfreezing of modules).
   • Tiled or partially untied weights can enforce local orthogonality and feature diversity.
   • Data augmentation and precise hyperparameter selection (e.g., interval length $\Delta t$, bit depth $N$) further enhance performance.

6. Cross-Domain Significance and Future Directions

Composite relative temporal encoding constitutes a unifying theme across domains requiring temporally aware, robust, and compositional representations. Its impact is notable in:

• Advancing state-of-the-art results in pose estimation, event-based recognition, and time series classification.
• Enabling biologically plausible associative memory models which afford structure-sensitive recall and learning.
• Facilitating efficient video understanding architectures that outperform dense attention models at markedly reduced computation and parameter cost.

A plausible implication is that further research may generalize composite relative temporal encoding to more abstract relational representations, such as learned graph relations or multi-modal compositional memories. Ongoing development of modular, interpretable neural architectures stands to benefit from the compositional and relational principles underlying these encodings. The continued trend is toward integrating biologically inspired mechanisms, data-driven architectural innovations, and rigorous empirical evaluations to improve both data efficiency and generalization.

7. Representative Quantitative Results

Domain	Composite Encoding Variant	Key Metric	Absolute Gain (Best vs. Baseline)
3D Pose (Shan et al., 2021)	$\mathcal{K}_P + \mathcal{K}_T$ + Multi-stage	MPJPE (mm)	30.1 mm vs 34.1 mm (−4.0 mm)
Time Series (Wang et al., 2015)	GAF + MTF	Test error (%)	Outperforms single-channel in 10/12
Event Vision (Innocenti et al., 2020)	Binary fusion ($N=8$, $\Delta t=$ 2.5ms)	Accuracy (%)	99.62% vs ≃97.75% prior
Video Enc. (Hao et al., 2024)	PoTGU/PoSTGU-channel grouped	Top-1 (%)	82.1% (Kinetics-400), lowest params

These results underscore the efficacy of composite relative temporal encoding as a foundational element for high-performance temporal modeling across disciplines.