Spatiotemporal Evaluation Pipeline

• Spatiotemporal evaluation pipeline is a structured framework designed to benchmark machine learning models on data with both spatial and temporal dependencies.
• The pipeline employs rigorous data perturbation and augmentation techniques, such as translation, rotation, scaling, and noise injection, to simulate real-world variations.
• It integrates diverse architectures—from 2D-CNNs to SlowFast networks—to ensure fairness, reproducibility, and robustness under controlled experimental protocols.

A spatiotemporal evaluation pipeline is a structured, multi-stage framework for rigorously benchmarking machine learning models on data exhibiting both spatial and temporal dependencies. These pipelines are designed to standardize model assessment for tasks where signals evolve across spatial domains and over time—foundational in domains such as video analysis, environmental monitoring, trajectory modeling, medical imaging, and additive manufacturing. A well-architected spatiotemporal evaluation pipeline ensures fairness, reproducibility, cross-method comparability, and robustness by precisely controlling data preprocessing, model interface conventions, augmentations, performance metrics, and reporting. Recent research in metal additive manufacturing defect detection exemplifies this approach, employing controlled data perturbations, systematic clip-generation strategies, and multi-model comparison to probe generalizability of deep spatiotemporal learners under realistic variabilities (Cherif et al., 2023).

1. Data Acquisition, Preprocessing, and Spatiotemporal Structuring

Spatiotemporal evaluation pipelines begin with the assembly and transformation of raw data streams to standardized clip-based representations. For example, in melt pool anomaly detection, high-speed cameras capture visible-light video at 2 kHz, producing RGB image sequences at 140×200 spatial resolution. Clips are constructed via sliding windows (typically S = 10 consecutive frames, covering ~4.65 ms of the melt pool solidification event), respecting both the spatial structure and the underlying temporal continuity (Cherif et al., 2023).

Model-specific input formatting is handled as follows:

• 2D-CNN Streams: Image sequences are reshaped from [N,S,C,H,W] to [N·S,C,H,W], relinquishing temporal ordering for feature extraction.
• 3D-CNN Streams: Sequences preserve their 5D tensor structure, [N,C,S,H,W], allowing temporal convolutions.
• Specialized Architectures: For two-pathway networks such as SlowFast, distinct temporal sampling rates and input channel striding (e.g., Fast pathway: 32 frames stride 2, Slow pathway: 4 frames stride 16) ensure complementary coverage of fine and coarse dynamics.

All architectures enforce normalization protocols, mapping pixel intensities to [0,1] and resizing frames to architecture-optimized spatial layouts (e.g., 227×227 for CaffeNet, 112×112 for R(2+1)D, and 224×224 for SlowFast). Label classes are encoded according to the semantic types of anomalies present in the task (e.g., Normal, Balling, Irregularity, Overheating).

2. Data Perturbation and Augmentation for Robustness Evaluation

To rigorously probe model generalization, the pipeline applies a sophisticated regimen of spatial and photometric augmentations that simulate real-world process variations. Perturbations are uniformly applied across each temporal clip and include:

• Translation: Offsets in X and Y axes up to ±25 pixels (5 px/step)
• Rotation: In-plane rotations θ ∈ {–25°, –20°, …, +25°}
• Contrast Scaling: Linear transformation I′ = c·(I–μ) + μ, with c ∈ [0.0, 0.9]
• Spatial Scaling: Downsampling/upsampling s ∈ [10%, 80%]
• Additive Noise:
  • Gaussian: I′=I + ϵ, ϵ∼𝒩(0,0.1)
  • Poisson: I′∼Poisson(I)

The resulting augmented dataset increases in size roughly 39-fold, furnishing a stringent environment for assessing robustness under various plausible degradations (Cherif et al., 2023).

3. Model Families and Spatiotemporal Integration Mechanisms

Spatiotemporal pipelines benchmark diverse neural architectures tailored to different integration strategies, each differing in how spatial and temporal dependencies are modeled:

• Two-Stream Networks (Simonyan & Zisserman archetype): Separate spatial (raw RGB) and temporal (multi-channel optical flow via small-RAFT) streams with CNN backbones (e.g., VGG16/ImageNet pretrain), fused at softmax or feature-map level.
• CNN-LSTM Hybrids: A per-frame spatial encoder (e.g., CaffeNet to fc6) followed by sequence processing via LSTM (hidden size 256), enabling explicit temporal memory.
• Factorized 3D CNNs (R(2+1)D): ResNet-18 blocks with 3D convolutional kernels split into spatial and temporal components, enabling efficient spatiotemporal feature disentanglement.
• SlowFast Networks: Dual-pathway architectures operating at different frame rates (α=8; e.g., 32 frames for Fast, 4 for Slow), with lateral fusion after each residual block; pretraining on large-scale video data (Kinetics400) enhances transfer and robustness.
• Baseline CNN1: A per-frame spatial baseline; serves as a lower bound for comparative assessment.

The pipeline enforces a consistent interface for all models—inputting S-frame clips, outputting four-way class predictions—and accommodates the architectural idiosyncrasies required by each method.

4. Training, Evaluation Protocols, and Robustness Assessment

Experimental validation protocol is meticulously controlled. For each model, the dataset is divided into fixed train/validation/test splits: three videos per class for training, one per class for validation, and the held-out validation set doubles as test depending on the training regime:

• (A) Train/validate on original, test on perturbed
• (B) Train/validate on perturbed, test on original
• (C) Train/validate on both, test on original and perturbed separately

Hyperparameters are explicitly recorded (SGD or Adam variants, batch sizes, epochs), ensuring reproducibility. Robustness is quantified as the accuracy drop (Δ) between clean and perturbed test sets.

The core pseudocode of the end-to-end pipeline encompasses dataset construction, model initialization, batched training with periodic validation, and post-training evaluation on test splits (detailed in the original source (Cherif et al., 2023)). No cross-validation is performed; the split is fixed.

5. Quantitative Metrics and Comparative Results

The principal evaluation metrics, computed across all runs, include:

• Accuracy:

$\mathrm{Acc} = \frac{TP + TN}{TP + TN + FP + FN}$

• F₁ Score:

$F_1 = \frac{2\,TP}{2\,TP + FP + FN}$

• Robustness Measure: Δ = Acc_clean – Acc_perturbed

Empirical findings reveal that baseline spatial models and standard spatiotemporal learners (CNN-LSTM, R(2+1)D) overfit to clean data but incur severe degradation (>60 percentage points) when evaluated on perturbed examples. In contrast, the pre-trained SlowFast network exhibits strong robustness, with average accuracy drops of only 8–10 percentage points in the most stringent settings (test_aug vs. test_clean), and maintains high overall accuracy when exposed to distributional shift (Cherif et al., 2023).

6. Pipeline Design Considerations and Broader Impact

The design and systematic validation of a spatiotemporal evaluation pipeline ensures that model selection, training protocols, and performance claims are substantiated under both nominal and challenging real-world conditions. By embedding richly parameterized augmentation strategies and enforcing uniform preprocessing and input shaping, these pipelines expose weaknesses in model generalization that would be masked in naively curated datasets. The findings that only large-scale video-pretrained, two-stream designs (specifically, SlowFast) retain performance under perturbation have direct implications for deploying defect detection in critical manufacturing—including cases where process deviations are subtle, transient, or distributed across spatial and temporal domains.

The rigid structuring and comprehensive reporting serve as a foundation for extending spatiotemporal evaluation to new domains, architectural innovations, and robustness-focused applications, mirroring best practices advocated in comparative studies across spatiotemporal prediction, video analysis, and anomaly detection in high-frequency sensor streams (Cherif et al., 2023).