T-REX: Vision-Based System for Autonomous Leaf Detection and Grasp Estimation

Abstract: T-Rex (The Robot for Extracting Leaf Samples) is a gantry-based robotic system developed for autonomous leaf localization, selection, and grasping in greenhouse environments. The system integrates a 6-degree-of-freedom manipulator with a stereo vision pipeline to identify and interact with target leaves. YOLOv8 is used for real-time leaf segmentation, and RAFT-Stereo provides dense depth maps, allowing the reconstruction of 3D leaf masks. These observations are processed through a leaf grasping algorithm that selects the optimal leaf based on clutter, visibility, and distance, and determines a grasp point by analyzing local surface flatness, top-down approachability, and margin from edges. The selected grasp point guides a trajectory executed by ROS-based motion controllers, driving a custom microneedle-equipped end-effector to clamp the leaf and simulate tissue sampling. Experiments conducted with artificial plants under varied poses demonstrate that the T-Rex system can consistently detect, plan, and perform physical interactions with plant-like targets, achieving a grasp success rate of 66.6\%. This paper presents the system architecture, implementation, and testing of T-Rex as a step toward plant sampling automation in Controlled Environment Agriculture (CEA).

T-REX: Vision-Based System for Autonomous Leaf Detection and Grasp Estimation

This paper introduces T-Rex, a gantry-based robotic system designed to automate the detection, selection, and grasping of plant leaves within greenhouse environments. The system integrates advanced vision-based techniques and motion planning algorithms to address inefficiencies in manual plant sampling for disease diagnosis. This work contributes to the growing field of Controlled Environment Agriculture (CEA) by offering a solution that leverages computer vision and robotics to improve operational efficiency in plant health monitoring.

Technical Overview

The T-Rex system incorporates a stereo vision pipeline, employing YOLOv8 for real-time leaf segmentation and RAFT-Stereo for depth estimation. This combination allows for the reconstruction of 3D leaf masks, facilitating accurate grasp point determination based on factors such as surface flatness, approachability, and edge margin. The system employs a 6-degree-of-freedom (6-DOF) gantry manipulator to execute leaf grasping using a custom microneedle-equipped end-effector, which simulates tissue sampling by clamping and penetrating the leaf surface.

Performance Metrics and Observations

The research reports a grasp success rate of 66.6% during trials with artificial plant models. This performance metric is noteworthy given the complexity of accurately identifying optimal grasp points without extensive training datasets. The paper details the mechanical design of the robot alongside an assessment of motion planning dynamics, highlighting the system's ability to select leaves using a Pareto optimization approach that balances clutter, distance, and visibility criteria.

Implications and Future Directions

The development of T-Rex marks an advancement toward fully automated crop health surveillance systems by integrating a closed-loop system architecture capable of interacting with plant surfaces to extract samples for genomic analysis. The exploration of hybrid approaches in perception and sampling exhibits promising potential for reducing manual labor in agricultural monitoring, which could have profound implications for large-scale production environments, particularly under cyber-physical systems in CEA.

Future work identified includes enhancing stereo camera resolution, refining segmentation models, and integrating real-time genomic analysis on-board the system. Potential adaptations to field robotics could extend the application of T-Rex to non-controlled environments, although challenges regarding navigation and movement stability need consideration.

Conclusion

T-Rex represents an essential contribution to agricultural robotics by demonstrating the feasibility of autonomous leaf detection and sampling. By integrating computer vision with pragmatic mission planning strategies and novel mechanical design, the system provides a foundational platform for subsequent innovations in plant disease diagnosis and precision agriculture. Its implementation within greenhouses suggests promising functionality, with further development required to translate the system into broader agricultural applications.