Internet-Scale Video-Action Dataset

• Internet-scale video-action datasets are expansive collections of annotated clips from platforms like YouTube, featuring diverse classes and actions.
• They employ automated mining, weak supervision, and consensus-driven labeling to efficiently scale annotation despite inherent noise.
• These datasets drive advancements in deep video representation learning, transfer learning, and multimodal recognition methodologies.

An Internet-scale video-action dataset is a large-scale corpus of videos and associated action labels, constructed by mining and annotating video data from public sources such as YouTube and other web platforms. These datasets offer broad coverage of human (and sometimes non-human) actions, events, or activities, and are foundational for training and benchmarking deep learning models in action recognition, localization, and multi-modal video understanding. Key properties include diversity of classes, vast quantity (hundreds of thousands to millions of clips), aggregation at the clip or segment level, metadata richness, and annotation protocols designed for both scalability and robustness to noise. Their proliferation has driven much of the progress in deep video representation learning.

1. Taxonomy and Characteristics of Internet-Scale Video-Action Datasets

These datasets are defined by quantities exceeding 100,000 clips (with many exceeding 1 million), broad class vocabularies (hundreds to thousands of action types), and heterogeneous data sources. Prominent exemplars include Kinetics-400/600/700 (Kay et al., 2017), Moments in Time (Monfort et al., 2018), HACS (Zhao et al., 2017), NoisyActions2M (Sharma et al., 2021), AVMIT (Joannou et al., 2023), ActionHub (Zhou et al., 2024), and Multi-Moments in Time (Monfort et al., 2019). Table 1 summarizes key comparison statistics based on the literature (Zhu et al., 2020):

Dataset	#Clips	#Classes	Clip Length	Label Style
Kinetics-400	306K	400	10 s	Single-label
Moments in Time	1M	339	3 s	Single-label
HACS Clips	1.5M	200	2 s	Single-label
NoisyActions2M	1.95M	7,098	Varied	Multi-label
Multi-Moments in Time	1.02M	292	3 s	Multi-label
AVMIT	57K (subset)	41	3 s	Audiovisual
ActionHub	3.6M desc.	1,211	—	Text, no video

Action vocabularies range from fine-grained human actions (Kinetics, HACS) to broad events including animals and phenomena (Moments in Time, Multi-Moments). Label styles include single-label (Kinetics), multi-label (Multi-Moments, NoisyActions2M), and multi-modal or description-enriched formats (ActionHub, AVMIT).

2. Construction Methodologies and Annotation Protocols

Dataset construction begins with class vocabulary design, often aggregating or filtering from prior benchmarks. Video sources include YouTube, Vimeo, Flickr, and curated footage sites (Kay et al., 2017, Monfort et al., 2018, Zhao et al., 2017, Zhou et al., 2024). Video retrieval is performed via title/metadata keyword queries or mining user-generated tags.

Candidate clip selection exploits weak supervision (image classifiers, heuristic keyword matches), followed by pre-filtering to reduce annotation burden. For example, Kinetics extracted candidate 10-second clips by running image-based action classifiers and selecting temporal windows around keyframes (Kay et al., 2017). HACS leverages both consensus and disagreement between two ResNet-50 image classifiers to select informative 2-second shots (Zhao et al., 2017).

Annotation procedures combine crowdsourcing (Amazon Mechanical Turk) for binary presence labels (Kay et al., 2017, Monfort et al., 2018) with protocol-driven aggregation (≥3/5 “Yes” for Kinetics, ≥75–85% consensus for Moments in Time). Some datasets use only weak web labels (NoisyActions2M: no human correction; all labels are ascribed from search context) (Sharma et al., 2021). Others, such as Multi-Moments in Time, extend single-label annotations with explicit multi-label judgments per video via iterative candidate generation and multiple annotator votes (Monfort et al., 2019).

Audio and multi-modal annotation is performed in datasets like AVMIT, with domain experts labeling both presence and prominence of an action’s audiovisual correspondence (Joannou et al., 2023).

3. Evaluation Protocols and Baseline Modeling

Official dataset splits consistently follow large-scale standards, with explicit train/validation/test assignments. Evaluation for single-label classification uses top-1 and top-5 accuracy: $$\text{Top-}k\;\mathrm{accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl(y_i\in \hat Y_i^{(k)}\bigr)$$ (Kay et al., 2017, Monfort et al., 2018). Multi-label tasks use mean Average Precision (mAP), Precision@K, and Recall@K metrics (Monfort et al., 2019). Temporal localization datasets (HACS Segments, ActivityNet) employ temporal Intersection over Union (tIoU) and mAP at multiple thresholds (Zhao et al., 2017).

Baseline architectures span frame-level 2D ConvNets + RNNs, Two-Stream networks (RGB/Flow fusion), 3D ConvNets (C3D, I3D), and more recently, multi-modal ensembles and audio-vision fusion models (Kay et al., 2017, Monfort et al., 2018, Zhao et al., 2017, Sharma et al., 2021). Self-supervised and contrastive pretraining, as well as distribution-balanced loss functions, have become prominent for handling noisy or weakly labeled datasets (Sharma et al., 2021).

4. Dataset Bias, Multimodality, and Hierarchical Semantics

Internet-scale datasets are inherently subject to label noise, class imbalance, and context/scene biases. For example, Kinetics assessed class-specific gender/age distributions and found some classes (e.g., “cheerleading,” “shaving beard”) to be demographically skewed, though no systematic performance bias was detected on minority groups (Kay et al., 2017). Moments in Time explicitly modeled auditory and visual classes, noting that 10-20% of its classes require audio for unambiguous recognition (Monfort et al., 2018), and AVMIT focused on curating high-quality audiovisual pairs for action localization (Joannou et al., 2023).

Some datasets, such as “Mining YouTube” (Kuehne et al., 2019), mitigate semantic inconsistency via a hierarchical action ontology (tree of meta/action subclasses), employing bottom-up and top-down probability refinement during both training and inference. This structure alleviates polysemy and improves inference in long-tailed, noisy data.

Zero-shot and open-vocabulary recognition are enabled in action-description corpora (ActionHub), where actions are paired with multiple human- or user-generated descriptions, filtered by semantic similarity to canonical action prototypes (Zhou et al., 2024).

5. Impact on Deep Action Recognition: Transfer Learning, Robustness, and Model Innovations

The scale and diversity of Internet-scale video-action datasets have transformed action recognition from 2D-CNNs with hand-crafted representations to large-scale pretraining of temporally and modally-rich neural architectures (Zhu et al., 2020). Transfer learning from these datasets yields demonstrable gains. For example, I3D models pretrained on HACS Clips achieve higher top-1 accuracy on UCF-101, HMDB-51, and Kinetics compared to Sports1M or Moments (Zhao et al., 2017).

Self- and weakly supervised methods—necessitated by label noise in datasets like NoisyActions2M—outperform standard supervised pretraining, conferring robustness to synthetic label corruption and video degradation (Sharma et al., 2021). The explicit multi-label structure of Multi-Moments in Time leads to better downstream mAP in challenging benchmarks (AVA, MultiTHUMOS, Charades) compared to single-label pretraining (Monfort et al., 2019).

Datasets encoding rich action descriptions (ActionHub) and aligned text-video embeddings are shaping state-of-the-art zero-shot action recognition frameworks, leveraging cross-modality and cross-action invariance objectives to generalize to previously unseen actions (Zhou et al., 2024).

6. Open Challenges and Frontiers

Remaining issues for Internet-scale video-action datasets include:

• Managing link rot and reproducibility challenges due to the periodic inaccessibility of web-sourced videos (Zhu et al., 2020).
• Reducing persistent scene and context bias, especially cases where background objects or settings dominate action cues.
• Constructing datasets with multiaction temporal annotation at fine temporal granularity, including support for simultaneous, sequential, or overlapping actions within untrimmed streams (Monfort et al., 2019, Zhao et al., 2017).
• Expanding domain diversity to account for cultural, geographic, and demographic variations, addressing privacy and anonymity constraints (e.g., masked face datasets) (Zhu et al., 2020).
• Developing methods for automated and scalable quality filtering of weak labels and mining hard negatives at scale (Kay et al., 2017, Zhao et al., 2017).
• Integrating advanced semantic hierarchies and multi-modal (audiovisual, text, context) annotation at scale, enabling robust zero-shot and compositional learning (Zhou et al., 2024, Joannou et al., 2023).

This suggests that future efforts will require continued automation in construction, new annotations for temporal and semantic complexity, and robust protocols for sustaining benchmark validity over time.

7. Summary Table: Representative Internet-Scale Video-Action Datasets

Name	#Clips	#Actions	Annotation	Modalities	Unique Features	Reference
Kinetics-400	306K	400	AMT majority	RGB, Flow	Diverse human actions, 10s, balanced	(Kay et al., 2017)
Moments in Time	1M	339	AMT consensus	RGB, Audio	Agents incl. animals, 3s, verb-centric	(Monfort et al., 2018)
HACS Clips/Segments	1.5M/139K	200	Visual, dense	RGB	2s/segment, dense temp. annotation	(Zhao et al., 2017)
NoisyActions2M	1.95M	7,098	Web metadata	RGB, text	High label noise, meta-rich	(Sharma et al., 2021)
Multi-Moments (M-MiT)	1.02M	292	Multi-label AMT	RGB, Audio	Multi-label per video	(Monfort et al., 2019)
AVMIT	57K	41	Lab, consensus	RGB, Audio	Audiovisual class curation, ready embeddings	(Joannou et al., 2023)
ActionHub	3.6M desc.	1,211	Description	Text	Rich text-video alignment for ZSAR	(Zhou et al., 2024)

All cited datasets and benchmarks have significantly advanced the methodology of action recognition, spatiotemporal representation learning, and multi-modal grounding at unparalleled scale.