Action-Part Semantic Relevance (APSR) Framework

• APSR is a framework for one-shot and zero-shot action recognition that integrates semantic body part relevance with spatio-temporal feature modeling.
• It computes relevance weights via embedding similarities between action descriptions and body parts, dynamically aggregating features for improved classification.
• Empirical results show enhanced accuracy on benchmarks through adaptive prototype construction and cross-modal refinement, highlighting its transferability.

The Action-Part Semantic Relevance-aware (APSR) framework is an architectural and algorithmic paradigm for human action recognition that explicitly integrates semantic part relevance into spatio-temporal modeling, particularly in data-scarce (“one-shot” or “zero-shot”) regimes. APSR exploits the interaction between high-level action semantics and low-level part dynamics, assigning contextual weights to body parts based on action descriptions, and adaptively aggregating features for robust classification.

1. Formal Problem Setting and Conceptual Strategy

APSR principally addresses one-shot and zero-shot 3D action recognition, in contexts where labeled exemplars per novel action class are extremely limited. Let the auxiliary set $\mathcal{D}_{\mathrm{aux}}$ comprise depth/skeleton videos from $C_{\mathrm{aux}}$ base classes, and the evaluation set $\mathcal{D}_{\mathrm{eval}}$ contains $C_{\mathrm{nov}}$ novel classes with minimal supervision (typically one exemplar per class). For each query video, the goal is to predict which novel class it belongs to, leveraging prior part-action semantics rather than direct feature learning on $\mathcal{D}_{\mathrm{eval}}$ (Liu et al., 2019).

The core APSR mechanism involves: (a) training a feature extraction backbone (often bidirectional spatio-temporal LSTM or graph convolutional network) on the base classes; (b) computing semantic relevance weights by mapping both action descriptions and part names into a shared embedding space (e.g., Word2Vec or CLIP) and measuring similarity; (c) aggregating part-wise visual features using these relevance weights to form the final discriminative representation; and (d) classification via nearest-neighbor or learned metric, using the single prototype per class.

2. Extraction and Representation of Action Parts

APSR models rely on a structured definition of body parts. For skeleton-based methods, each frame $t$ of a video sequence is represented as joint coordinates $\mathbf{x}_{pt} \in \mathbb{R}^3$ for $P$ anatomical joints ($P=25$ for NTU RGB+D 120) (Liu et al., 2019). A bidirectional spatio-temporal LSTM is employed: each unit $(p,t)$ receives input $\mathbf{x}_{pt}$ and its context along both spatial (joint order) and temporal axes. The LSTM produces hidden states $\mathbf{h}_{pt}^{\rightarrow}$ and $\mathbf{h}_{pt}^{\leftarrow}$, yielding concatenated part features $$\mathbf{F}_{pt} = [\mathbf{h}_{pt}^{\rightarrow}; \mathbf{h}_{pt}^{\leftarrow}] \in \mathbb{R}^{2H}$$.

For image-based methods, parts are defined as head, torso, arms, hands, and legs (five regions). Semantic part actions (e.g., “hand waving”, “head drinking”) are predicted per region using specialized ResNet classifiers, obtained after localization via a semi-FCN (Zhao et al., 2016).

DynaPURLS extends APSR to dynamic partitioning: a cross-modal attention module affinely queries skeleton nodes with part-aware text embeddings, adaptively grouping joint-temporal nodes to maximize correspondence with hierarchical semantic descriptions (Zhu et al., 12 Dec 2025).

3. Semantic Relevance Computation and Integration

Semantic relevance is quantified via textual embedding similarity. Given an action description $a$ (“wield knife towards other person”) and part name $p$ (“right hand”), APSR computes $E(a)$ and $E(p)$ in the embedding space ($\mathbb{R}^{300}$ for Word2Vec; $\mathbb{R}^d$ for CLIP). Relevance $R(a,p)$ is defined as:

$$R(a,p) = \max \Big(0, \frac{E(a)\cdot E(p)}{\|E(a)\|\|E(p)\|} \Big)$$

Normalized part weights are then $s_{a,p} = R(a,p)/\sum_{u=1}^P R(a,u)$, forming a distribution over parts for each action (Liu et al., 2019). In DynaPURLS, GPT-3 or similar LLMs generate structured descriptions for the action’s global movement, subparts, and temporal segments, producing multi-scale semantic anchors. Cross-modal attention explicitly aligns each anchor to skeleton visual features (Zhu et al., 12 Dec 2025).

In image-based APSR, discriminative parts for each action are selected via Linear Discriminant Analysis of part-action feature variances, but the concept can be generalized to a learned relevance matrix or attention mechanism (Zhao et al., 2016).

4. Network Architectures and Training Objectives

APSR system architectures include three principal components:

• Part Encoder: ST-LSTM or Shift-GCN aggregated over spatial/temporal axes, producing part-wise embeddings (Liu et al., 2019, Zhu et al., 12 Dec 2025).
• Semantic Relevance Module: Off-line embedding lookup and relevance computation, or (in advanced forms) hierarchical text encoding via CLIP/GPT-3 (Zhu et al., 12 Dec 2025).
• Classification Head: For skeleton-base, auxiliary weighted cross-entropy loss $$\mathcal{L}_{\mathrm{cls}} = -\sum_{(x,c)\in\mathcal{D}_{\mathrm{aux}}} \sum_{p=1}^P \sum_{t=1}^T s_{c,p} \log \sigma_c(\mathbf{F}_{pt})$$ encourages focus on relevant parts (Liu et al., 2019). For end-to-end DynaPURLS, visual-textual alignment is optimized by symmetric InfoNCE loss across global, part, and temporal anchors, weighted by learned $\alpha$ (Zhu et al., 12 Dec 2025).

In APSR for images, final action classification fuses global and part-action features selected for discriminability, via weighted concatenation and linear SVM (Zhao et al., 2016). Incorporating part selection via learned relevance is suggested as a generalization.

5. Inference, Prototype Construction, and Adaptive Refinement

For one-shot recognition, APSR aggregates the part-wise features of an exemplar $\Omega$ weighted by relevance $s_{c_{\mathrm{nov}},p}$, yielding a prototype:

$$f(\Omega; S_{c_{\mathrm{nov}}}) = \sum_{p=1}^P \sum_{t=1}^T s_{c_{\mathrm{nov}},p}\ F_{pt}(\Omega)$$

For a query $x$, the same weighted aggregation is computed and cosine similarity to each prototype determines the predicted class (Liu et al., 2019). In DynaPURLS, textual anchors themselves are adaptively refined at test time by an affine transformation:

$F' = \mathrm{Norm}( S \odot F + \Delta F )$

Confidence-aware memory bank and pseudo-labelling ensure robust adaptation to unseen class distributions at inference (Zhu et al., 12 Dec 2025).

6. Empirical Results and Benchmark Comparisons

APSR on NTU RGB+D 120, one-shot 3D action recognition yields superior accuracy over baselines (average pooling, fully connected, attention network): 45.3% vs. 42.9%/42.1%/41.0%, with performance declining sharply for smaller auxiliary sets (down to 29.1%) (Liu et al., 2019). DynaPURLS achieves state-of-the-art zero-shot and generalized zero-shot accuracy on NTU 60/120 and PKU-MMD, surpassing prior art by margins up to 14.5% (e.g., 89.06% vs. 74.53%) (Zhu et al., 12 Dec 2025).

In image-based action recognition, semantic part-action fusion via SVM outperforms global-only recognition by up to 3.8% mAP on major benchmarks (Zhao et al., 2016).

Method	Benchmark	Accuracy / mAP (%)
APSR (One-shot)	NTU RGB+D 120	45.3
Avg. Pooling	NTU RGB+D 120	42.9
DynaPURLS (Zero-shot)	NTU RGB+D 120	89.06
SVM Part Fusion	VOC 2012	86.4
ResNet-50 (bbox)	VOC 2012	82.7

7. Features, Limitations, and Future Directions

APSR frameworks incorporate semantic priors that are shown to generalize better to novel actions than learned attention weights, which often overfit to base classes. Text embedding quality and description granularity exert a significant impact; ambiguous descriptions compromise relevance assignments. Skeleton-only systems cannot model object-centric actions unless appearance features are also integrated, and granularity is limited by joint definitions, possibly missing fine manipulations (Liu et al., 2019, Zhu et al., 12 Dec 2025).

Prospective advances include fusing RGB and depth modalities, incorporating object detectors, meta-learning for relevance refinement, leveraging contextual text encoders (BERT, GPT), and extending APSR to few-shot regimes beyond single exemplars (Liu et al., 2019, Zhu et al., 12 Dec 2025).

A plausible implication is that APSR paradigms—by unifying multi-scale semantic decomposition, dynamic part aggregation, and test-time adaptive refinement—provide a foundation for robust, transferable action recognition across both 2D and 3D domains, particularly under limited supervision. This suggests that future work will further explore hierarchically structured and attention-based mechanisms to model complex human–object–part interactions, potentially improving transferability and interpretability in action understanding.