Neural-Behavioral Representation of Natural Whole-body Movement in Monkeys

Abstract: Understanding how cortical activity represents natural whole-body behaviors in primates remains challenging. Limited by the diversity of movements and inaccessibility of large-scale neural representation of whole-body kinematics, previous motor decoding studies focused on constrained tasks and limited limb movements. Here, we present a neural-behavioral recording and modeling framework for freely moving monkeys, combining large-scale epidural cortical signals from distributed sensory- and motor-related areas with synchronized multi-view motion capture through a custom-made data collection platform. We reconstructed whole-body monkey kinematics and learned a compact behavior prior using an autoregressive encoder-decoder model. Conditioned on neural signals, the model decoded accurate and realistic whole-body movement without explicit physical constraints. Our results provide a novel proof-of-concept approach for decoding natural whole-body movements in primates using large-scale intracranial neural activity.

• The paper presents a framework for decoding natural whole-body movement from large-scale epidural signals in freely moving monkeys.
• It employs a transformer-based neural encoder and a variational autoregressive decoder to predict 10-second movement sequences with significantly lower ADE and FDE errors.
• Experimental results show superior long-horizon accuracy and physical plausibility compared to LSTM-based decoders and behavioral priors.

Neural-Behavioral Decoding of Natural Whole-Body Movement in Monkeys

Introduction

Understanding how cortical dynamics encode behaviorally rich, naturalistic, whole-body movements is an open challenge in systems neuroscience and brain-machine interfaces (BMIs). Prior research in primate motor decoding has focused on highly constrained tasks, typically involving repeated, isolated limb movements, with neural recordings from limited cortical regions. These approaches, often using discriminative regression or classification, do not scale to the high-dimensional, temporally coherent movements found in unconstrained, naturalistic behavior. The study "Neural-Behavioral Representation of Natural Whole-body Movement in Monkeys" (2605.29355) establishes a framework for the direct reconstruction of continuous, physically plausible, whole-body kinematics from large-scale epidural cortical signals in freely moving macaques, leveraging synchronized multi-view motion capture and a variational autoregressive generative modeling approach.

Experimental Platform and Data Acquisition

A robust platform was implemented for simultaneous large-scale epidural cortical recording and high-resolution whole-body 3D kinematic capture in a freely moving Macaca fascicularis. Distributed epidural electrodes (62 channels) spanning bilateral sensorimotor and parietal cortices were chronically implanted. The neural data acquisition was synchronized with video capture from an octagonal, eight-camera RealSense D435i system within a custom-designed enclosure, reconstructing 3D skeletons at 10 Hz after downsampling.

Figure 1: Pipeline for decoding natural whole-body movement from epidural neural signals—showing the hardware setup, motion capture, and neural-behavioral modeling pipeline.

This design ensured minimal movement-induced artifacts due to the skull-fixed epidural approach, maximizing spatial brain coverage while mitigating the invasiveness associated with classic ECoG arrays. Multi-hour datasets were collected across diverse spontaneous behaviors including walking, climbing, swinging, and transitions to and from rest.

Kinematic Behavioral Representation

A 20-marker kinematic model (15 primary "joints") with analytical, fixed-length limb segments allowed for a compact, informative behavioral state vector $P \in \mathbb{R}^{152}$ at each time step. The state comprised root translation and rotation, local 3D joint positions, rotations (in sin/cos Euler representations), and velocities. High-fidelity reconstruction of raw 3D poses from this state demonstrated mean errors typically below 1 cm, ensuring the representation's suitability for generative modeling and neural decoding tasks.

Neural-Behavioral Generative Modeling Framework

The core computational contribution is a variational sequence model for movement, conditioned on concurrent epidural neural features. The framework incorporates:

• Transformer-based neural encoder: Inputs are time-spectral neural features (multi-taper PSD and Morlet wavelets, 1–150 Hz, 62 channels), projected with channel-wise positional embeddings and a learned global aggregation token.
• Behavioral encoder: Recent behavioral states are encoded via a two-layer MLP.
• Mixture-of-experts decoder: Six expert MLPs and a gating network decode autoregressive transitions in the behavioral latent space, conditioned on both neural and prior state context.
• Variational regularization: Latent conditioning variables (dimension 16) ensure a non-degenerate posterior, mitigating posterior collapse.

Training is staged to incrementally shift from teacher-forced ground-truth to purely autoregressive prediction, stabilizing long-horizon dynamics.

Figure 2: Framework of Neural-Behavioral Model—the neural encoder, behavior prior, and variational autoregressive decoder conditioned on neural and prior state information.

Comparative Evaluation and Quantitative Results

The study evaluates prediction against standard LSTM-based neural decoders and pure behavioral priors. The main findings include:

• Multi-second movement reconstruction: The model autoregressively predicts 10 s sequences from a single initial state and associated neural trajectory.
• Superior long-horizon accuracy: Average Displacement Error (ADE) and Final Displacement Error (FDE) for all joints are markedly lower for the Neural-Behavioral Model (ADE_horizontal: 6.97 cm, ADE_vertical: 6.18 cm) compared to LSTM and prior-only models. Pearson $r$ correlations exceed 0.4 joint-wise, versus $<$0.2 for behavioral prior and $<$0.1 for LSTM.
• Temporal scale robustness: At short horizons ($<$1 s), the behavior prior approximates short, plausible sequences; over longer horizons, the neural-conditioned model retains physical consistency and behavioral specificity, reflected in reduced error growth and stable $r$.
• Kinematic expressiveness: The model accurately reconstructs diverse, context-dependent behaviors including static, locomotor, swing, and climbing actions, while failure cases localize to subtle out-of-distribution actions without sufficient neural discrimination.
• Ablative contrast: Exclusion of neural features results in stochastic, non-specific movement sampling; LSTM-based directionless decoding (even with ground-truth velocity directions) fails to achieve physically plausible coordination.

  Figure 3: Visualization of reconstructed whole-body movement by three models during a swinging movement—highlighting the high spatiotemporal fidelity of the Neural-Behavioral Model versus priors and LSTM.

  

  Figure 4: Model performance (FDE and $r$) across time horizons and movement axes—demonstrating statistically significant decoding improvements of the Neural-Behavioral Model for both vertical and horizontal actions.

Additional analyses (see Figs. 8–10) show the Neural-Behavioral Model's lower FDE along dominant movement axes for all evaluated movement types, and its resilience to compounding error in autoregressive unrolling.

Discussion and Implications

This work directly establishes that large-scale epidural field potentials distributed across sensorimotor-parietal cortices suffice for decoding high-dimensional, realistic primate movement under unconstrained, naturalistic conditions. The use of a neural-conditioned prior over behavior (rather than direct coordinate regression) is essential for temporal coherence and physical plausibility in high-dimensional output spaces.

Contradicting prior claims that only invasive microelectrode or ECoG signals could enable kinematic decoding at this scale, the results demonstrate that epidural signals with sufficient spatial distribution and frequency-bandwidth, when coupled with suitable generative approaches, form a viable substrate for natural movement brain-machine interfacing.

These insights have direct implications for next-gen neuroprosthetics, BMIs for full-body exoskeletons, and ethologically valid studies of neural coding. The released, synchronized datasets further serve as a benchmark for cross-modal representation learning endeavors.

Future Directions

The study acknowledges that generalization to inter-individual, inter-session, and more complex motor behaviors requires larger, more heterogeneous datasets and further advances in neural-behavioral modeling (e.g., cross-modal pretraining, data augmentation, alignment regularization). Extensions to non-human primates and translation to human clinical settings are tenable but will require robust signal normalization and unsupervised cross-domain alignment.

Conclusion

This study delivers an authoritative demonstration of decoding naturalistic, full-body primate kinematics from large-scale epidural fields via generative, neural-conditioned sequence modeling. It sets numerical performance baselines and methodological standards for future research at the intersection of computational neuroscience, machine learning, and neuroengineering, advancing the tractable reconstruction of unconstrained movement from neural population codes.