Wheelchair Mobile Manipulator Systems

• Wheelchair-based mobile manipulators are integrated robotic platforms that combine powered wheelchair mobility with dexterous arms, multimodal sensing, and autonomy.
• Systems range from high-speed athletic implementations using specialized hardware to assistive devices employing vision-based teleoperation and shared autonomy.
• Key challenges include vision-control latency, dynamic stability, and force control, with future research targeting high-frequency feedback and unified kinodynamic planning.

A wheelchair-based mobile manipulator is an integrated robotic platform that combines the mobility of a powered or manually configured wheelchair with the dexterity of a robotic manipulator arm and, in modern systems, a rich multimodal sensing, planning, and autonomy stack. These systems enable a range of tasks from athletics (e.g., robotic wheelchair tennis) to assistive activities of daily living, leveraging control, perception, and safety architectures tailored for joint mobility and manipulation in human-centered environments (Zaidi et al., 2022, Abuduweili et al., 2024, Dai et al., 2024).

1. System Architectures and Component Choices

Wheelchair-based mobile manipulators use diverse hardware arrangements depending on application domain. Representative systems include:

• Athletic Manipulators (ESTHER): Built around a motorized Top End Pro Tennis wheelchair with dual brushless DC motors (ODrive D6374), combined with a 7-DoF Barrett WAM arm for dynamic groundstrokes. The architecture integrates high-rate perception (six stereo rigs, Velodyne Puck LiDAR), centralized ROS-based computation, and a regulation tennis racket as end effector (Zaidi et al., 2022).
• Human-Assistive Manipulators (WeHelp): Utilizes the Hello Robot Stretch RE1, comprising a mobile base, slender linear-lift manipulator with a gripper, onboard depth camera and microphone (with external high SNR replacement), under Ubuntu/ROS. The manipulator performs fetch, door opening, and augments user autonomy with teleoperation/remote control (Abuduweili et al., 2024).
• Dynamic Balancing Manipulators (Ballbot with Wheelchair): CMU’s ballbot couples two torque-sensing 7-DoF arms to drive a wheelchair via physically compliant manipulation. RealSense T265, internal IMU and encoders, and high-frequency wrench estimation complete the sensing stack (Dai et al., 2024).

Integrated safety is realized through multi-layered E-stop systems (wireless, manual, watchdog) and conservative actuation bounds. The choice of manipulator (e.g., industrial arm vs. lightweight domestic) and mobile base (differential drive vs. balancing) tunes the system for high speed, payload, or compliance as needed.

2. Modeling: Kinematics and Dynamics

Manipulator and Base Kinematics

• Serial Arm Kinematics: The Barrett WAM is modeled with standard homogeneous transforms, with joint-space $\boldsymbol{q}\in\mathbb{R}^7$ and end-effector pose

$$T_0^7(\boldsymbol{q}) = \prod_{i=1}^7 A_i(\theta_i) \in SE(3)$$

where $A_i$ follows the Denavit-Hartenberg convention (Zaidi et al., 2022).

• Differential Drive Wheelchairs: Modeled by

$$v = \frac{v_R+v_L}{2},\qquad \omega = \frac{v_R-v_L}{b}$$

with standard planar robot dynamics

$$m\dot v = F_{\rm drive} - F_{\rm fric},\quad I_z\dot\omega = \tau_{\rm drive,R}-\tau_{\rm drive,L} - \tau_{\rm fric}$$

• Composite Base–Arm State: Full system configuration:

$x_{\text{sys}} = [x\ y\ \phi\ q_1 \dots q_7]^T$

enabling coordinated placement of the manipulator in workspace given wheelchair mobility.

• Nonholonomic and Balancing Systems: Ballbot with wheelchair employs planning in the reduced nonholonomic state $\dot q_w = [v_x, \omega]^T$, and closed-loop compliance for safe physical human-robot interaction (Dai et al., 2024).

Dynamic Interaction Modeling

• Arm Lagrangian Dynamics: For each joint,

$M(q)\ddot q + C(q, \dot q)\dot q + g(q) = \tau$

• Ball–Racket Impact (Athletics): Inelastic approximation:

$$v_z^+ = -e v_z^-,\quad e\approx 0.7;\qquad J_z = m_b(1+e)v_z^-$$

Impact timescale $\sim2\,\text{ms}$ (Zaidi et al., 2022).

• Whole-Body Push Interaction: For balancing robots maneuvering external carts,

$$M_w(q_w)\ddot q_w + C_w(\dot q_w)\dot q_w = \Gamma_w - N_w(\dot q_w)$$

with $\Gamma_w$ (force and torque) shaped via body lean and mapped through end-effectors (Dai et al., 2024).

3. Perception, Sensing, and State Estimation

Vision Pipelines

• Athletic Contexts: Multi-camera stereo (ZED2 rigs) with color-based thresholding, morphological processing, and volumetric 3D triangulation to determine ball positions at up to 150 Hz. Depth error covariance is modeled per rig as $\sigma_z^2(D) = a D^2 + b D + c$, tuned experimentally (Zaidi et al., 2022).
• Assistive Contexts: Visual person-tracking uses YOLO-v3-tiny on RGB-D camera streams at 2–3 Hz. Wheelchair user is identified via bounding-box area and height in COCO “person” detections (Abuduweili et al., 2024).

Filtering and Fusion

• Athletic Manipulators: Extended Kalman Filter (EKF) fuses per-rig ball positions into a shared 6D state $x_b = [X, Y, Z, \dot X, \dot Y, \dot Z]^T$, with model-based ballistic rollout for interception prediction (Zaidi et al., 2022).
• Balancing Systems: Fusion of wheelchair pose/velocity (RealSense T265, encoders) with wrench estimates provides both kinematic and inertial state for online adaptation (Dai et al., 2024).

4. Control, Planning, and Compliance

Base and Arm Controllers

• Differential Drive Bases: Global planning typically via graph search (Dijkstra over occupancy maps); local with Timed Elastic Band (TEB); velocity servoing by wheelwise PID (Zaidi et al., 2022).
• Manipulation: Barrett WAM and similar arms use onboard PID for joint trajectory following:

$$\tau_i = K_{p,i}(q_i^d - q_i) + K_{d,i}(\dot q_i^d - \dot q_i)$$

Optional impedance control law is available, not enabled in the baseline.

• Whole-Body Pushing (Ballbot): Quasi-static optimization problems are solved to set body lean angles $\Phi$ required for a desired wheelchair velocity, subject to force and torque limits:

$$\min_{\Phi} (\mathbf{A}\Phi - \boldsymbol f)^T Q (\mathbf{A}\Phi - \boldsymbol f) + \Phi^T R \Phi$$

This enforces both velocity-tracking and compliance/safety (Dai et al., 2024).

Shared Autonomy and Modes

• WeHelp: Control is orchestrated by speech-based state machines that initiate YOLO-based following (with discrete corrective actions), teleoperation (Xbox controller mapping to base/arm), and browser-mediated remote mode. No PID or path planning is used in assistive following mode (Abuduweili et al., 2024).
• Planning for Athletic Execution: Motion plans for stroke execution select three-via-point swings, timing execution $t_{\rm trigger}=t_{\rm impact}-T_{\rm swing}$ to maximize racket head velocity at ball arrival (Zaidi et al., 2022).

Physical Compliance

• Impedance Control: Arms are modeled with variable $\bar M_d, \bar B_d, \bar K_d$ parameters ensuring soft, safe responses to external disturbances.
• Ballbot Base Compliance: PID+PD for lean tracking, with disturbance-rejection bandwidth tuned so that both shocks and human “bumps” are absorbed safely (Dai et al., 2024).

5. Experimental Evidence and Evaluation

Benchmark Metrics

A comparison of key empirical results from published systems is as follows:

Platform	Max Lin. Vel. (m/s)	Racket/Gripper Lin. Vel. (m/s)	Typical Task Success (%)	Notable Features
ESTHER (Zaidi et al., 2022)	4.34–10 (Manual)	10 (racket)	53%–93% hit, 33%–80% full success (tennis)	Full ROS, stereo rigs, pro wheelchair
WeHelp (Abuduweili et al., 2024)	Not specified	Not specified	~100% tracking (behind-fwd ≤1 m/s)	Modular autonomy modes, Stretch RE1
Ballbot+WC (Dai et al., 2024)	0.45	N/A (push)	<10% error in velocity tracking	Online inertial ID & compliant push

• ESTHER: Racket-head speed achieves up to 10 m/s, hit rates 73% (court, 7.9 m), 60% (12.8 m), with prediction errors in (X,Y,Z) <0.20 m; wheelchair repositioning tested up to 2 m, with success dropping for reposition distances >1.5 m due to vision pipeline latency (Zaidi et al., 2022).
• WeHelp: In following mode, 100% user tracking at ≤1 m/s (behind), but <50% for side tracking above 0.2 m/s; task times for novices to fluent users ranged 40–90 s for ADLs such as chair moving and door opening (Abuduweili et al., 2024).
• Ballbot Manipulator: Velocity tracking rise times ~1.1–1.7 s (linear/angular) for loaded vs. empty wheelchair; compliance validated by a 30% damped lateral response and “human-like” smoothness (acceleration norm within 19–25% of skilled human) (Dai et al., 2024).

6. Challenges, Limitations, and Future Directions

Practical Limitations:

• Vision–Control Latency: In high-speed applications, per-camera latencies (~100 ms) and EKF convergence can delay critical base positioning, reducing reachability/success in time-critical athletic tasks. For wheelchair tennis, late convergence limits effective repositioning to ~1.5 m (Zaidi et al., 2022).
• Planning Modularity: Many pipelines plan for base and manipulator sequentially. Coupled kinodynamic planners for dynamic, torque-limited whole-body behavior are required for further agility.
• Force Control: High-dynamic impacts (robot racket/ball at $\sim$10 ms scale) strain conventional position control. Pure position control is inadequate for robust interaction under these extremes; future directions include high-rate impedance or hybrid force/position wrist controllers (Zaidi et al., 2022).
• Teleoperation Ergonomics and Reliability: For assistive ADL robots, controller ergonomics are currently a barrier for users with significant upper-limb limitations (Abuduweili et al., 2024). Ubuntu/ROS stability issues (e.g., freezing in remote control) and fixed camera rates constrain responsiveness.

Technical Constraints:

• Dynamic Limits: Systems using quasi-static planners (e.g., ballbot+wheelchair) inherently limit maximum speed and responsiveness. Absence of explicit center-of-pressure or support-polygon constraints may compromise stability on uneven terrain (Dai et al., 2024).
• Manipulation Generality: Few systems perform autonomous grasp-point detection; most rely on prior knowledge or direct teleoperation for grasping and manipulation (Abuduweili et al., 2024, Dai et al., 2024).

Opportunities and Directions:

• Integration of high-rate vision (≥500 Hz multi-camera fusion), explicit spin estimation (athletic robots), dynamic stability modeling in planners (ballbot), and integration of learning from demonstration for complex task adaptation are principal directions cited for research enhancement.
• Increased use of sampling-based kinodynamic planning, higher-bandwidth actuation (series elastic, pneumatic), and tactile/force sensing will extend task competence and safety.
• Extension to unstructured environments—ramped/wheeled access, crowded navigation, multi-robot cooperative tasks (e.g., doubles tennis, hospital beds)—will require higher-level reasoning, intent inference, and workspace awareness (Zaidi et al., 2022, Abuduweili et al., 2024, Dai et al., 2024).

7. Research Significance and Outlook

Wheelchair-based mobile manipulators concretely advance the merger of mobile robotics, dexterous manipulation, and interactive autonomy. ESTHER establishes the first reproducible, open-source baseline for athletic human-scale robots engaging in regulation tennis. WeHelp demonstrates that commercial platforms and off-the-shelf AI perception (YOLO, Deep Speech) can be robustly composed for multi-modal shared autonomy in domestic tasks. Ballbot-controlled wheelchair manipulators validate advanced whole-body planning, online system identification, and impedance-based compliance for physically interactive tasks.

Core research challenges remain: scaling base–arm coordination for dynamic tasks, robustifying perception to vision-limited and cluttered scenes, and achieving human-level motion fluency at pro-level velocities. A plausible implication is that future systems will converge toward unified, reinforcement-learned, kinodynamically coupled architectures, with high-frequency force/vision feedback, thereby paralleling human sensorimotor control, while guaranteeing safety and reliability in complex human environments.

---

References:

• "Athletic Mobile Manipulator System for Robotic Wheelchair Tennis" (Zaidi et al., 2022)
• "WeHelp: A Shared Autonomy System for Wheelchair Users" (Abuduweili et al., 2024)
• "Wheelchair Maneuvering with a Single-Spherical-Wheeled Balancing Mobile Manipulator" (Dai et al., 2024)