emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation

Abstract: Hands are the primary means through which humans interact with the world. Reliable and always-available hand pose inference could yield new and intuitive control schemes for human-computer interactions, particularly in virtual and augmented reality. Computer vision is effective but requires one or multiple cameras and can struggle with occlusions, limited field of view, and poor lighting. Wearable wrist-based surface electromyography (sEMG) presents a promising alternative as an always-available modality sensing muscle activities that drive hand motion. However, sEMG signals are strongly dependent on user anatomy and sensor placement, and existing sEMG models have required hundreds of users and device placements to effectively generalize. To facilitate progress on sEMG pose inference, we introduce the emg2pose benchmark, the largest publicly available dataset of high-quality hand pose labels and wrist sEMG recordings. emg2pose contains 2kHz, 16 channel sEMG and pose labels from a 26-camera motion capture rig for 193 users, 370 hours, and 29 stages with diverse gestures - a scale comparable to vision-based hand pose datasets. We provide competitive baselines and challenging tasks evaluating real-world generalization scenarios: held-out users, sensor placements, and stages. emg2pose provides the machine learning community a platform for exploring complex generalization problems, holding potential to significantly enhance the development of sEMG-based human-computer interactions.

• The paper presents the emg2pose benchmark, a large-scale sEMG dataset with 193 users, 370 hours of data, and 29 kinematic categories.
• It details two core tasks, pose regression and pose tracking, and introduces competitive models including the novel vemg2pose.
• The study highlights generalization challenges with complex hand kinetics and suggests advanced machine learning techniques for personalized model improvement.

An Overview of "emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation"

The paper "emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation" introduces the emg2pose benchmark, which aims to advance the development of universal surface electromyography (sEMG)-to-pose models. The primary motivation revolves around the limitations of existing computer vision-based hand-tracking systems and the potential of sEMG as a viable alternative for robust hand pose estimation.

Dataset and Capabilities

The emg2pose dataset addresses the challenges posed by user anatomy, sensor placement, and hand kinematics that complicate universal sEMG-to-pose model development. The dataset is notable for its size and diversity, featuring 193 users, 370 hours of data collection, and 29 diverse kinematic categories. This extensive dataset significantly surpasses existing sEMG benchmarks, enabling a variety of experimental evaluations. It was collected using a high-fidelity 16-channel sEMG device paired with a 26-camera motion capture system, ensuring high-quality pose labels.

Experimental Tasks

The dataset supports two core tasks: pose regression and pose tracking. Pose regression focuses on predicting hand joint angle sequences from sEMG signals, a partially observable task given the unknown initial hand pose and velocity. Conversely, the pose tracking task provides this initial pose information, thereby reducing observability challenges. These tasks serve to catalyze research progress in scenarios where computer vision is impractical.

Model Baselines and Architectures

The paper provides three competitive baseline models: NeuroPose, SensingDynamics, and the novel vemg2pose model. While NeuroPose and SensingDynamics operate by predicting joint angles directly, vemg2pose employs a velocity-based method that predicts joint angular velocities. This approach is significant for its autoregressive nature and effective use of Time-Depth Separable Convolutions (TDSs), which enable parameter-efficient feature extraction.

Performance Evaluation

The experiments underscore vemg2pose’s superior performance over NeuroPose and SensingDynamics, particularly in scenarios involving held-out users and stages. These results highlight the challenges associated with generalization across previously unseen users and kinematic categories, emphasizing the dataset’s role in aiding the development of robust sEMG models.

Discussion of Generalization Challenges

This research identifies anatomical differences and complex dynamic gestures as significant variables influencing model performance. Generalization is specifically challenging for motions involving intricate hand kinematics and interactions, commonly problematic for vision-based systems. The paper's analysis suggests that increasing dataset scale in terms of training users and diverse stages improves model performance, validating the strategic emphasis on dataset extensiveness.

Future Research Directions

The authors advocate for exploration of state space and diffusion-based methods to model sequence-level data, reflecting an open invitation to apply more sophisticated machine learning techniques that have been successful in analogous domains. Furthermore, addressing sensor and anatomical variability with probabilistic modeling techniques is presented as a promising avenue. The paper suggests that overcoming the existing performance gap with computer vision systems requires these innovative approaches, potentially extending to personalized model training.

Conclusion

In conclusion, the emg2pose benchmark provides a substantial and necessary resource for advancing sEMG-based hand pose estimation. Offering a large-scale, diverse dataset alongside competitive baseline models, this work lays the groundwork for future experiments that explore complex generalization problems. This contribution holds promise for enhancing the practicality and accuracy of human-computer interaction technologies utilizing sEMG signals. The dataset and framework are likely to catalyze further methodological advancements in the use of biosignals within machine learning contexts.