Pose-TGCN: Spatiotemporal Graph ConvNets

• Pose-TGCN is a neural architecture that integrates spatial and temporal relationships by constructing an augmented spatiotemporal graph for human pose modeling.
• It employs graph convolutional operations combined with sequence-aware self-attention to jointly capture body structure and motion dynamics.
• The approach has demonstrated improved 3D pose prediction accuracy across datasets, enhancing applications in video understanding and motion synthesis.

Pose-based Temporal Graph Convolutional Networks (Pose-TGCN) are neural architectures that integrate spatial and temporal relationships among human body joints for pose forecasting and motion analysis. By constructing an augmented spatiotemporal graph structure where nodes represent joints at specific time frames and edges encode both skeletal connections and temporal continuity, Pose-TGCN efficiently captures the dynamics of human motion sequences. Recent advances, such as the Multi-Graph Convolution Network (MGCN), extend this paradigm by fusing spatial, temporal, and sequence-aware attention mechanisms to enhance 3D pose prediction accuracy in challenging, realistic motion-capture scenarios (Ren et al., 2023).

1. Spatiotemporal Graph Construction

Pose-TGCN relies on a comprehensive graph representation where each node corresponds to a specific joint at a particular time step, for a total of $T \cdot V$ nodes given $T$ frames and $V$ joints per frame. The edges in this framework are designed to encode two fundamental relationships:

• Spatial (Skeletal) Edges: Within each frame, joints are connected following the skeleton's physical topology. The spatial adjacency matrix $A_s \in \mathbb{R}^{V \times V}$ is constructed using multiple-hop partitions $g_0, ..., g_D$ based on graph distance on the skeleton tree:

$$(g_k)_{ij}= \begin{cases} 1, & d(i, j) = k \ 0, & \text{otherwise} \end{cases}$$

$A_s = \sum_{k=0}^D g_k$

• Temporal Edges: For each joint, temporal adjacency $A_t \in \mathbb{R}^{T \times T}$ connects the same joint across nearby frames within a temporal window $L$:

$$(A_t)_{t_1, t_2} = \begin{cases} 1, & |t_1 - t_2| \leq L \ 0, & \text{otherwise} \end{cases}$$

• Augmented Adjacency Matrix: These are composed into a large block matrix $A \in \mathbb{R}^{(T\cdot V) \times (T\cdot V)}$, where each block encodes intra-frame spatial relationships and inter-frame temporal dependencies. The matrix includes normalization by the graph Laplacian:

$$\widehat{A} = D^{-\tfrac{1}{2}} A D^{-\tfrac{1}{2}}$$

where $D$ is the degree matrix with $D_{ii} = \sum_j A_{ij}$ (Ren et al., 2023).

2. Graph Convolution Operations

The core of Pose-TGCN is the propagation and transformation of joint-state features across this spatiotemporal graph using graph convolutional networks (GCN). At each layer $l$, the node feature matrix $X^{(l)} \in \mathbb{R}^{(T\cdot V) \times F_l}$ is updated as:

$X^{(l+1)} = \sigma(\widehat{A} X^{(l)} W^{(l)})$

or in the multi-partition form,

$$X^{(l+1)} = \sigma\left(\sum_{k=0}^D \widehat{A}_k X^{(l)} W^{(l)}_k\right)$$

where $\sigma$ is a nonlinear activation (typically ReLU), and $W^{(l)}$ (or $W^{(l)}_k$ per partition) are learnable weight matrices (Ren et al., 2023).

This process enables simultaneous modeling of body structure and dynamics, in contrast to previous methods that handled temporal and spatial domains separately (e.g., LSTM+GCN hybrids).

3. Sequence-Aware Attention Integration

Beyond static graph convolutions, Pose-TGCN architectures employ sequence-aware self-attention to improve long-range temporal modeling:

• Q/K/V Streams: Three parallel MGCN branches generate query, key, and value tensors $Q, K, V \in \mathbb{R}^{T \times V \times C}$:

$$Q = \mathrm{MGCN}_Q(X_{in}, A), \quad K = \mathrm{MGCN}_K(X_{in}, A), \quad V = \mathrm{MGCN}_V(X_{in}, A)$$

• Pseudo-Autoregressive Attention: A lower-triangular mask restricts attention to past and present frames, forcing the model to output per-step 3D joint displacements.
• Anchor-based Attention: Attention scores are computed via masked softmax, and future pose predictions are expressed as convex combinations of anchor poses from previous frames. For each coordinate $d$ of joint positions,

$$X_{T+i}^d = \sum_{k=1}^T \lambda_{ik}^d \; \text{anchor}_k^d, \quad \sum_k \lambda_{ik}^d = 1, \quad \lambda_{ik}^d \geq 0$$

with anchors and weights $\lambda$ computed from the outputs of the MGCN branches and attention scores (Ren et al., 2023).

4. Full Network Architecture and Training Protocol

The canonical Pose-TGCN pipeline employs:

• MGCN Backbone: An input tensor $X_{in} \in \mathbb{R}^{T \times V \times 3}$ is passed through a stack of 4–6 MGCN layers, with main branch widths (e.g., 3→64→32→64→3) and narrower configurations in attention streams.
• Temporal Alignment (TCN Head): A one-dimensional temporal convolution head extends output length from $T$ to $T+K$ frames to accommodate sequence prediction.
• Spatial-Temporal Refinement: A final MGCN refines predictions across the spatiotemporal domain.
• Loss Function: The Mean Per-Joint Position Error (MPJPE) is used,

$$\mathcal{L}_{\mathrm{MPJPE}} = \frac{1}{VK} \sum_{k=T+1}^{T+K} \sum_{v=1}^V \left\| \hat{x}_v^k - x_v^k \right\|_2$$

where $\hat{x}_v^k$ and $x_v^k$ denote predicted and ground-truth 3D positions, respectively.

• Optimization: Training uses Adam, batch size 256, initial learning rate 0.1 with staged decay at epochs 20, 35, and 45, for a total of 50 epochs (Ren et al., 2023).

5. Empirical Evaluation and Benchmarks

Pose-TGCN and its MGCN instantiation have been evaluated on standard 3D human pose forecasting datasets:

Dataset	Protocol	Metric	SOTA Prior (STS-GCN)	MGCN (SAA-10)
Human3.6M	10 input frames, predict 80/400/1000 ms ahead	MPJPE (mm)	10.1 / 38.3 / 75.6	8.6 / 34.4 / 83.6
AMASS→3DPW	Generalize 400ms / 1000ms	MPJPE (mm)	24.5 / 42.3 (3DPW)	24.0 / 40.3

MGCN consistently outperforms the prior best STS-GCN across prediction horizons and datasets, indicating the advantage of concurrent spatial-temporal modeling and attention. The model is evaluated using 22 joints (Human3.6M), 18 joints (AMASS), and tested on 3DPW for cross-dataset robustness (Ren et al., 2023).

6. Relation to Prior Art and Variants

Prior approaches to human pose forecasting often relied on autoregressive models (LSTM variants, Transformer encoders) or hybridized GCN+RNN architectures, each handling temporal and spatial dependencies separately. These approaches suffered from issues such as vanishing/exploding gradients (RNNs) and limited inter-joint communication across time. Pose-TGCN, and specifically the MGCN variant, alleviates these limitations by jointly encoding body structure and motion within a unified graph, and by supplementing convolution with sequence-aware attention for enhanced context aggregation (Ren et al., 2023).

7. Practical Impact and Prospects

Pose-TGCN architectures enable robust prediction of complex human motion, supporting applications in video understanding, action recognition, and motion synthesis. The demonstrated generalization across datasets (e.g., AMASS→3DPW) suggests effective learning of natural motion priors. A plausible implication is that further integration of multi-partite graph or attention mechanisms may continue to advance accuracy and robustness, especially for long-term sequence forecasting and occlusion recovery.