Spatial-Temporal Weighted Contrastive Loss

• Spatial-Temporal Weighted Contrastive Loss is a technique that decomposes and weights spatial and temporal features to enhance discriminative embeddings in dynamic data.
• It integrates multi-level contrastive objectives with InfoNCE losses and attention mechanisms to capture fine-grained intra- and inter-class distinctions.
• Its applications span action recognition, traffic forecasting, and link prediction, consistently improving performance over baseline models.

Spatial-Temporal Weighted Contrastive Loss (STWCL) constitutes a class of contrastive learning objectives that leverage explicit decomposition, weighting, or synchronization of spatial and temporal components in feature representations. This approach is central to recent advances in spatiotemporal representation learning, particularly in domains such as action recognition, traffic forecasting, and dynamic link prediction. STWCL methods are characterized by their exploitation of fine-grained intra- and inter-class distinctions across both spatial and temporal axes, frequently incorporating learned attention, memory banks, and hierarchical multi-view architectures to construct robust and discriminative embeddings (Zhang et al., 2023, Li et al., 2023, Tai et al., 2024).

1. Mathematical Formulation and Core Loss Structures

Prominent STWCL frameworks define loss terms as sums of weighted contrastive objectives over spatial and temporal branches or views, often further decoupled into node-/edge-/time-level contrastive losses in graph-based settings.

For example, in skeleton-based action recognition, the STD-CL (Spatial–Temporal Decoupling Contrastive Learning) formulation models spatial- and temporal-specific features $s, t \in \mathbb{R}^D$, each subjected to an InfoNCE loss over class-conditioned positives and negatives maintained in memory banks. The total loss is: $$\mathcal{L}_{\text{total}} = -\sum_{k} y_{i,k}\log p_{i,k} + \mathcal{L}_{\text{NCE}}^{\text{spa}} + \mathcal{L}_{\text{NCE}}^{\text{tem}}$$ with each branch-specific InfoNCE loss given by: $$\mathcal{L}_{\text{NCE}}^{\text{spa}} = - \sum_{s^+ \in P_{\text{spa}}} \log \frac {\exp({\langle s, s^+ \rangle/\tau})} {\exp({\langle s, s^+ \rangle/\tau}) + \sum_{s^- \in N_{\text{spa}}} \exp({\langle s, s^- \rangle/\tau})}$$ and an analogous expression for the temporal branch (Zhang et al., 2023).

In multi-view spatiotemporal graph contexts, as in CLP and STS-CCL, the total loss aggregates supervised objectives with spatial-temporal contrastive components: $$\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{main}} + \lambda_1\,\mathcal{L}_{\mathrm{Node}} + \lambda_2\,\mathcal{L}_{\mathrm{Edge}} + \lambda_3\,\mathcal{L}_{\mathrm{Time}}$$ where each $\mathcal{L}_{\cdot}$ term represents InfoNCE-based contrastive losses over distinct structural levels (Tai et al., 2024, Li et al., 2023).

These STWCL formulations utilize explicit weighting parameters ($\lambda$, $\epsilon$, $\alpha$) to tune relative contributions from spatial, temporal, semantic, or augmentation-specific branches.

2. Spatial and Temporal Feature Decomposition

STWCL advances are predicated on the explicit decoupling or modeling of spatial and temporal features. In STD-CL, given a tensor $X \in \mathbb{R}^{J\times T\times C}$ (joints × frames × channels), spatial-specific features are obtained via temporal mean pooling: $X_s = \frac{1}{T} \sum_{t=1}^T X_{:, t, :}$ while temporal-specific features use spatial mean pooling: $X_t = \frac{1}{J} \sum_{j=1}^J X_{j, :, :}$ These are then linearly projected, concatenated, and further mapped into a shared embedding space for contrastive metric learning (Zhang et al., 2023).

In heterogeneous graphs (CLP), spatial decomposition proceeds via two complementary schemas: node-level and edge-level structural views, each constructed with type- or relation-specific graph neural networks and weighted pooling layers. Temporal decomposition leverages parallel recurrent channels (LSTM for long-term, GRU for short-term dynamics) to encode the evolution of node representations across time steps (Tai et al., 2024).

3. Attention and Weighting Mechanisms

STWCL methodologies frequently employ both explicit (learned) and implicit (pooling-based) attention to weight spatial or temporal elements. In STD-CL, attention arises from two sources: uniform pooling operations (effectively uniform attention) and learned linear projections ($W_\theta, W_\phi, W_\zeta, W_\sigma$) that scale joint, frame, or channel contributions in the embedding projections (Zhang et al., 2023).

In CLP, attention is formalized using multi-head attention weights at both node and edge aggregation levels: $$\alpha^{r t}_{a,b} = \frac{\exp(\beta^{r t}_{a,b})}{\sum_{k\in\mathcal{N}^{r t}_a} \exp(\beta^{r t}_{a,k})}$$ and

$$\delta_a^{r t} = \frac{\exp(\gamma_a^{r t})}{\sum_{r'} \exp(\gamma_a^{r' t})}$$

These allow heterogeneous importance weighting over edge-types and temporal relations (Tai et al., 2024).

In STS-CCL, semantic contextual negative filtering is performed using a combination of connectivity adjacency and Jensen-Shannon semantic similarity, enabling negative sample selection with explicit spatial and semantic criteria (Li et al., 2023).

4. Augmentation and Mutual-View Schemes

STWCL approaches reinforce representation robustness via augmentation. In STS-CCL, two augmentation pipelines—basic (edge/attribute masking, temporal fusion) and strong (learned dynamic view selection via Gumbel-softmax)—produce mutual views. Contrastive losses are then computed across these paired augmentations, and are further synchronized using a spatial-temporal synchronous contrastive module (STS-CM) (Li et al., 2023).

Mutual-view prediction, where one augmentation is used to predict the future representation of another, enforces invariance and cross-view consistency, implemented via paired InfoNCE loss terms in the STS-CCL objective: $L_{mvp} = L_{sts}^b + L_{sts}^s$ (Li et al., 2023).

5. Hyperparameters, Implementation, and Ablation

STWCL performance is sensitive to several key hyperparameters:

• Projection dimension $D$, channel reduction rate $r$, InfoNCE temperature $\tau$, and counts for hard-positive/negative sampling in memory bank formulations (e.g., $N_H^+, N_H^-, N_R^-$ in STD-CL) (Zhang et al., 2023).
• Relative weighting of spatial, temporal, and semantic contrastive losses ($\lambda_1, \lambda_2, \lambda_3$, $\epsilon$, Top-$u$ for negative filtering) (Li et al., 2023, Tai et al., 2024).
• Augmentation strengths (masking rates, fusion coefficients, Gumbel temperature) that control the invariance/difficulty balance for contrastive signal (Li et al., 2023).

Empirical ablation underscores that spatial-only or temporal-only contrastive branches yield smaller improvements than the full joint STWCL strategy. For instance, STD-CL improves top-1 accuracy on NTU-120 (X-sub) from 84.9% (baseline) to 85.6% (+0.7); with level-wise boosts most pronounced on medium-difficulty classes (Zhang et al., 2023). In dynamic graph prediction, CLP with explicit loss weighting achieves 10.10% AUC and 13.44% AP improvements relative to state-of-the-art baselines across diverse datasets (Tai et al., 2024).

6. Domains of Application and Empirical Outcomes

STWCL objectives are adopted in:

• Skeleton-based action recognition, where spatial/temporal decoupling resolves ambiguous motions by enforcing tight intra-class clustering and maximal inter-class separation at both joint and frame levels (Zhang et al., 2023).
• Urban traffic forecasting, where STS-CCL’s synchronous and semantic contextual contrastive losses compensate for label-scarcity and enhance node-level distinction (Li et al., 2023).
• Link prediction in temporal heterogeneous networks, with CLP’s node, edge, and time-level contrasts bridging spatial and temporal heterogeneity to outperform prior GNN and transformer-based baselines (Tai et al., 2024).

Performance gains are consistently attributed to the multi-level, weighted, and domain-attuned structure of STWCL frameworks.

7. Comparative Summary of Major Frameworks

Model / Domain	Spatial-Temporal Decomposition	Attention/Weighting	Key Results
STD-CL / Skeleton Action (Zhang et al., 2023)	TemporalMean and SpatialMean pooling; memory banks	Learned linear projections; uniform pooling	+0.7% Top-1 on NTU-120 (X-sub)
STS-CCL / Traffic Forecast (Li et al., 2023)	STS-CM (ProbSparse attention + DI-GCN); node-level contrast	mutual-view and semantic-context weights ($\epsilon$, Top-$u$)	Best MAE at $\epsilon\approx0.7$
CLP / Link Prediction (Tai et al., 2024)	Node-/Edge-level GAT, LSTM/GRU for time	Multi-head self-attention, learned $\lambda_i$	Avg. +10.10% AUC, +13.44% AP

STWCL approaches integrate fine-grained, level-adaptive contrastive losses across space and time, with attention and explicit weighting tuned for specific application requirements. Empirical studies show that such weighting and structural granularity are essential for maximizing the discriminative power and generalization of spatiotemporal representation learning models.