Image-Based Visual Servoing (IBVS)

• Image-Based Visual Servoing (IBVS) is a control approach that regulates pose and trajectory using real-time image feature feedback without requiring explicit 3D reconstruction.
• IBVS combines classical geometric feature extraction with deep learning techniques to enhance robustness and accuracy in dynamic and unstructured settings.
• IBVS employs mathematical tools like pseudoinverse control laws and Lyapunov-based stability analysis to ensure convergence despite challenges like occlusions and depth estimation errors.

Image-Based Visual Servoing (IBVS) is a foundational paradigm in vision-based robot control in which task objectives are defined and regulated directly in terms of image-space features acquired from onboard cameras. This approach establishes a visual feedback loop from pixel measurements to robot actuation, enabling closed-loop regulation of pose, trajectory, and interaction with dynamic environments. IBVS is characterized by its reliance on the real-time extraction, processing, and control of visual features—bypassing the need for explicit 3D scene reconstruction or position estimation—and is central to manipulation, grasping, mobile robotics, aerial navigation, dynamic object interception, and advanced human–robot interaction.

1. Mathematical Principles and Control Law Formulation

The canonical IBVS structure treats the vector of observed image features as the core feedback signal. Let $s(p)\in\mathbb{R}^{2N}$ be the stacked vector of $N$ image-plane keypoints $p_i=[u_i,v_i]^T$ (pixel coordinates), and let $s^*=s(p^*)$ denote the target configuration. The visual error is $e=s(p)-s^*$. The evolution of $s$ under camera motion is governed by the image-Jacobian ("interaction matrix") $L(s, Z)$, which relates the camera’s spatial velocity $$\mathbf{V}_c=[v_x,v_y,v_z,\omega_x,\omega_y,\omega_z]^T$$ to the rate of change of $s$: $\dot{s} = L(s, Z) \, \mathbf{V}_c$ where $L(s, Z)$ is block-diagonal, with each block $J_p(p_i, Z_i)\in\mathbb{R}^{2\times6}$ admitting a detailed expression as a function of image coordinates, focal length, and feature depth $Z_i$. Classical IBVS adopts a pseudoinverse-based control law: $\mathbf{V}_c = -\lambda L^+(s, Z) \, e$ where $\lambda>0$ is a gain and $L^+$ denotes the Moore–Penrose pseudoinverse. Actuation commands are computed via the robot's geometric Jacobian: $\dot{q}=J_\text{robot}^{-1}(q)\,\mathbf{V}_c$, enabling task-space velocities to be recast into joint velocities or other platform-appropriate commands. The framework is extensible to moment-based features, similarity transforms, or more abstract representations for specialized tasks (Amiri et al., 2024).

2. Visual Feature Extraction, Representation, and Learning

Feature extraction in IBVS is fundamental to system performance. Traditional methods employ robust geometric primitives (e.g., points, lines, SIFT keypoints, or image moments) with various detection and matching protocols to ensure repeatability and invariance in dynamic scenes (Haviland et al., 2020, Wang et al., 22 Sep 2025). In contemporary systems, deep learning methods, notably convolutional neural networks (CNNs), are leveraged for keypoint localization and feature regression to overcome the fragility and limited expressiveness of classical detectors. For example, a modified VGG-19 with average pooling and adaptive learning rates has been employed to regress the precise pixel coordinates of task-relevant object corners, with data augmentation and k-fold cross-validation ensuring generalization and robustness (MAE $\lesssim0.015$ pixels on validation) (Amiri et al., 2024). Two-stream CNNs directly learn a mapping from image pairs to task-space pose errors and velocity commands, bypassing analytic Jacobian estimation (Liu et al., 2019). Hybrid pipelines integrate global template matching, Lucas-Kanade alignment in feature (VGG) space, and recurrent GRU-based predictors to ensure occlusion robustness and sub-pixel tracking under difficult conditions (Lee et al., 29 Oct 2025).

3. Depth, Image-Jacobian Conditioning, and Stability

An enduring challenge in IBVS is the dependence of the interaction matrix on accurate depth estimates for all visual features. Approaches diverge based on the sensing configuration:

• Monocular IBVS: Depth per feature is estimated heuristically, approximated as constant, or pre-computed using CAD/stereo for structured environments. Fixed depth assumptions are often sufficient for local convergence if feature geometry is well conditioned (e.g., non-collinear points) (Haviland et al., 2020, Amiri et al., 2024).
• Stereo IBVS: Direct measurement of depth via stereo disparity allows for more accurate Jacobian computation. However, overdetermined formulations (e.g., 3D–2D constraints from multiple features in a 6-DoF system) can introduce local minima where the error lies in the nullspace of $L$. Feedforward–feedback architectures, utilizing joint-space feedforward actions with adaptive Youla parameterized feedback, guarantee avoidance of these local minima and global asymptotic stability (Li et al., 12 Jun 2025).
• Dense/photometric IBVS: Student’s t-mixture modeling of full-image photometry allows IBVS to operate directly on pixel intensities, with derivative-based interaction matrices. Heavy-tailed distributions provide robustness against outliers and partial occlusions (P et al., 2020).

Stability analyses generally employ Lyapunov-based techniques or exploit the exponential decay induced by the proportional control law. For overdetermined or adaptive architectures, composite Lyapunov functions and invariance principles prove convergence of both image error and joint/pose error, even under model uncertainty (Li et al., 12 Jun 2025, Li et al., 11 Jun 2025, Zhang et al., 2024).

4. Extensions: Adaptation, Occlusion Avoidance, and Predictive Control

Advanced IBVS frameworks address operational complexities in real-world tasks:

• Adaptive, Model-Independent, and Uncalibrated IBVS: Real-time online estimation of the image Jacobian via finite-difference or least-squares update allows for uncalibrated or disturbed robot/camera configurations. Feature detection via deep learning and direct Jacobian regression further reduce the reliance on calibrated models (Yin et al., 2023).
• Occlusion Avoidance and Safety Constraints: Control barrier function (CBF) theory enables safe closed-loop IBVS under probabilistic occlusion and uncertainty, defining forward-invariant sets in image space (e.g., via distance-to-obstacle in image plane) and ensuring chance-constrained operation. These constraints can be embedded within model predictive control (MPC) for real-time satisfaction of both servoing and safety objectives (Zhang et al., 2023).
• Visual Servoing under Delay: High-speed applications (e.g., drone interception) are constrained by sensor-processing latency. Delayed Kalman Filter (DKF) observers predict the current feature state, allowing for increased control bandwidth and reduction of terminal tracking error (CEP improvement from 0.457 m to 0.089 m in multicopter interception) (Yang et al., 2024, Yan et al., 2024).

5. Application Domains: From Manipulation to Aerial and Mobile Systems

IBVS is pervasive in manipulation (eye-in-hand systems, grasping under unobservable depths, dual-arm cooperation), mobile ground robotics, and aerial systems (quadrotor navigation, multicopter interception, UAV landing, gate passage) (Haviland et al., 2020, Amiri et al., 2024, Zhang et al., 2024, Wang et al., 22 Sep 2025, Yang et al., 2024, Rungta et al., 2020). In dynamic scenes, IBVS is augmented with online object detection, segmentation, and keypoint regression, as well as hybrid or learned modules for handling partial observations and occlusions. For underactuated vehicles (e.g., quadrotors), reduced-dimension analytic formulations and neuro-analytical pipelines have enabled stable high-speed control, markerless tracking, and efficient onboard inference (e.g., single-path 1.7M–param ConvNets yielding $\sim$500 Hz inference rates) (Mocanu et al., 26 Jul 2025).

Table: IBVS Control Law Variants for Selected Applications

Application Domain	Feature Type	Control Law
Robotic Manipulation	Keypoints, CNN corners	$V_c = -\lambda L^+ e$; joint velocity via $J^{-1}$ (Amiri et al., 2024)
Quadruped Manipulator	Spherical centroid	MPC on sphere; super-twisting observer for $v_T$ (Zhang et al., 2023)
UAV Interception	LOS pixel error	SO(3) barrier-Lyapunov outer loop; IBVS in camera (Yang et al., 2024)
Dual-arm Cooperation	AprilTag corners	Joint-space steepest descent on visual+pose cost (Zhang et al., 2024)

6. Limitations, Open Problems, and Future Directions

IBVS effectiveness is inherently linked to feature observability, Jacobian conditioning, robustness to occlusion/outlier events, and latency in perception-action loops. Limitations include:

• Sensitivity to depth approximation; degenerate/invariant feature configurations (e.g., collinearity) can induce singularities or poor convergence.
• Overdetermined image-Jacobian mappings in stereo or multi-feature settings, risking convergence to undesired local minima—addressed via feedforward–Youla parameterization and joint-space planning (Li et al., 12 Jun 2025).
• Robustness to occlusion, lighting variation, and environmental complexity, requiring hybrid deep-feature tracking, sequential estimation, and learning-based feature interpretation (Lee et al., 29 Oct 2025, P et al., 2020).
• Generalization and performance under high-speed or agile maneuvering where image-projection dynamics are tightly coupled to vehicle orientation and full-body control, necessitating integrated NMPC, spherical projection, and aggressive visibility-constraint enforcement (Qin et al., 2022, Qin et al., 2021, Qin et al., 2022).

Emergent lines of research focus on: adaptive and markerless IBVS, sim-to-real transfer with knowledge distillation, learning robust latent visual representations, and embedding chance-constrained barrier function theory into predictive control for safety and tractability.

---

References:

(Amiri et al., 2024, Haviland et al., 2020, Wang et al., 22 Sep 2025, P et al., 2020, Mocanu et al., 26 Jul 2025, Lee et al., 29 Oct 2025, Yan et al., 2024, Li et al., 12 Jun 2025, Li et al., 11 Jun 2025, Yin et al., 2023, Zhang et al., 2023, Liu et al., 2019, Yang et al., 2024, Qin et al., 2022, Zhang et al., 2023, Zhang et al., 2024, Rungta et al., 2020, Qin et al., 2021)