MIDI Score to Robotic Motion

• The paper demonstrates a direct, real-time mapping from MIDI files to robotic joint commands, bypassing pre-defined motion templates.
• The system integrates collision-aware motion planning and expressive control to accurately emulate musical dynamics and articulation.
• Residual reinforcement learning is employed to optimize bowing and fingering trajectories, enhancing sound quality and physical efficiency.

A MIDI score to robotic motion pipeline comprises a computational and mechanical architecture that translates MIDI-based musical instruction directly into expressive, collision-aware movement trajectories executed by a robotic instrument, usually for the purpose of physically performing music with accurate timing, dynamics, and articulation. Such systems are critical for robotic musicianship, especially for string instruments demanding precise control of kinematic variables such as bow angle and force, and are increasingly evaluated for their musicality via human-blind assessment protocols modeled on the “Musical Turing Test” paradigm (Sudhoff et al., 7 Jan 2026).

1. Technical Overview of the MIDI-to-Motion Pipeline

The MIDI-to-motion pipeline is an end-to-end system that receives MIDI representation of a musical score and outputs real-time, physically viable joint commands for a multi-degree-of-freedom (DoF) robotic platform. The primary input, MIDI, encodes note-onsets, offsets, dynamics (velocity), pitch, and potentially other control parameters, but lacks direct physical meaning for actuation tasks. The pipeline comprises a parsing and representation stage, musical feature analysis, motion generation (often with embedded collision and kinematic constraints), and a robotic control phase that executes the trajectory on the hardware.

In recent work, this pipeline was developed for a UR5e-based robotic cellist, employing the Universal Robot Freedrive mode to facilitate learning and execution of bowing maneuvers without the need for an external motion capture system. The core innovation is direct mapping from symbolic score to motor execution, bypassing both the need for pre-defined bowing templates or human-motion traces and enabling real-time “sight-reading” (Sudhoff et al., 7 Jan 2026). The system records ground-truth joint actuation data during performances via Real-Time Data Exchange (RTDE), yielding a labeled dataset for benchmarking and further research.

2. System Architecture and Computational Methods

Fundamental to the pipeline is the translation from discrete MIDI events to continuous, high-frequency robotic joint trajectories. This involves:

• Parsing and Temporal Alignment: MIDI files are parsed and temporally quantized (typically at sub-beat resolution, e.g., 1/16th notes), aligning music events with a high-rate trajectory planning grid.
• Motion Planning: Each note-onset, pitch, and dynamic parameter is mapped to a corresponding physical gesture—e.g., initiating a bow stroke at a specific angle and velocity, modulating string contact force to achieve desired loudness and timbral characteristics. Collision-awareness is embedded into the trajectory planning procedure to avoid self-intersection and instrument damage.
• Expressive Control: The system must generate not just correct notes but expressive qualities (attack time, micro-dynamics, articulation). This often involves parameterization of bowing motion—angle, force, travel speed—based on the musical context, sometimes drawing on human-labeled performance data for calibration.
• Control and Execution: Final trajectories are converted to joint-space commands and delivered to the robot over high-speed control channels (e.g., RTDE). The Universal Robot Freedrive allows for interactive, kinesthetic teaching and error reduction, supporting human-like expressivity (Sudhoff et al., 7 Jan 2026).

3. Evaluation via Musical Turing Test Protocols

Assessment of a pipeline’s musicality typically invokes “Musical Turing Test” paradigms, where human listeners attempt to discriminate between robot- and human-generated performances in double-blind, forced-choice setups. In the referenced pipeline, performances generated by the system, as well as human recordings of the same scores, were presented to 132 listeners, with the identity of the performer hidden. Such setups yield empirical data on perceptual indistinguishability and thus measure whether the robotic musician achieves a level of expressive realism comparable to human performance.

Earlier work in related domains (e.g., "Neural Translation of Musical Style" (Malik et al., 2017), "Echoes of Humanity" (Figueiredo et al., 29 Sep 2025), and "If Turing played piano with an artificial partner" (Dotov et al., 2024)) consolidate the role of forced-choice, controlled and randomized trials, and mixed-methods analysis—including both quantitative accuracy and qualitative perceptual coding—in evaluating the success of computational-to-performance pipelines under human scrutiny.

4. Data Collection and Benchmarking

A notable feature of the UR5e cello system is systematic joint-level data acquisition during performance, facilitated by RTDE. The resulting dataset offers fully labeled correspondence between auditory outputs (recorded cello sound) and joint trajectories for a canonical set of five standard repertoire pieces (Sudhoff et al., 7 Jan 2026). Human reference recordings are distributed alongside robotic outputs, forming an open benchmark for reproducibility and future comparative research.

Providing such paired datasets enables both benchmarking and the development/training of advanced learning algorithms for musical control—including supervised learning, inverse kinematics estimation, and reinforcement learning approaches.

5. Learning and Optimization Strategies

To improve beyond static, engineered control policies, the pipeline introduces a residual reinforcement learning (RL) framework. In this context, residual RL is deployed atop a baseline deterministic controller (e.g., traditional kinematics with heuristic music-mapping) and incrementally refines control actions based on feedback, such as acoustic quality or efficiency of string-crossing. This hybridization allows the system to discover subtle, task-specific optimizations in bowing and fingering trajectories beyond the reach of conventional planning.

The incorporation of RL serves dual objectives: (1) optimize for acoustic objectives (sound quality, note precision); (2) enhance physical efficiency and robustness (e.g., minimizing extraneous motion). A plausible implication is that such methods will be essential for matching or exceeding human expressivity, particularly as repertoires and instrument complexity increase (Sudhoff et al., 7 Jan 2026).

6. Context, Limitations, and Future Directions

The described pipeline eliminates dependence on motion-capture-based imitation and advances the state of the art for real-time, score-to-motion robotic musicianship. However, its performance ceiling is subject to mechanical and control bandwidth limitations inherent to current robot arms, as well as challenges in autonomous adaptation to new musical styles or extended techniques.

Future directions proposed by the authors emphasize improving string-crossing efficiency, sound quality, and extending the approach to more complex instruments and ensemble contexts. Broader implications include the opportunity to employ advanced style transfer, robustly integrate real-time sensory feedback, and potentially use human-robot paired datasets for multimodal generative modeling and interactive co-performance (Sudhoff et al., 7 Jan 2026).

7. Relation to Broader Research and Benchmarks

The emergence of robust, data-driven MIDI-to-motion pipelines coincides with trends in expressive performance modeling, autonomous music generation, and human-robot musical interaction. Prior efforts in purely virtual domains—such as deep neural translation of musical style (Malik et al., 2017), Turing-style auditory indistinguishability trials for AI-music (Figueiredo et al., 29 Sep 2025), and real-time improvisational duet paradigms (Dotov et al., 2024)—frame analogous evaluation challenges.

The release of paired robotic-human performance datasets and the formalization of robotic musical Turing Tests for physical machines establishes, for the first time, a concrete, reproducible benchmark for physical expressivity and realism in robotic musicianship (Sudhoff et al., 7 Jan 2026). This suggests a convergence of symbolic-to-physical music translation, interactive learning, and objective human evaluation as the central research axes for the next generation of autonomous performance robotics.