Comprehensive Robotic Cholecystectomy Dataset (CRCD): Integrating Kinematics, Pedal Signals, and Endoscopic Videos

Abstract: In recent years, the potential applications of machine learning to Minimally Invasive Surgery (MIS) have spurred interest in data sets that can be used to develop data-driven tools. This paper introduces a novel dataset recorded during ex vivo pseudo-cholecystectomy procedures on pig livers, utilizing the da Vinci Research Kit (dVRK). Unlike current datasets, ours bridges a critical gap by offering not only full kinematic data but also capturing all pedal inputs used during the procedure and providing a time-stamped record of the endoscope's movements. Contributed by seven surgeons, this data set introduces a new dimension to surgical robotics research, allowing the creation of advanced models for automating console functionalities. Our work addresses the existing limitation of incomplete recordings and imprecise kinematic data, common in other datasets. By introducing two models, dedicated to predicting clutch usage and camera activation, we highlight the dataset's potential for advancing automation in surgical robotics. The comparison of methodologies and time windows provides insights into the models' boundaries and limitations.

• The paper introduces a comprehensive dataset capturing synchronized kinematics, pedal signals, and stereo endoscopic videos during robotic cholecystectomy procedures.
• It employs data from da Vinci Research Kit procedures on pig livers, featuring 60 fps video and detailed sensor signals for accurate 3D reconstructions.
• Preliminary results indicate high accuracy in predicting surgeon actions using RandomForest classifiers, paving the way for surgical automation advancements.

Introduction

The integration of AI and machine learning techniques into Minimally Invasive Surgery (MIS) has opened up new opportunities for the development of data-driven surgical tools. However, existing datasets often lack comprehensive recordings that include kinematic data, or detailed information on the movement and position of surgical instruments. The Comprehensive Robotic Cholecystectomy Dataset (CRCD) aims to fill this gap by providing an extensive collection of data from robotic cholecystectomy procedures performed on pig livers, including kinematics, pedal signals, and stereo endoscopic videos.

Dataset Composition

The CRCD dataset is derived from procedures performed using the da Vinci Research Kit (dVRK), and includes foot pedal signals and timestamps for synchronizing video and kinematic data. The videos obtained from the endoscope cameras are recorded at 60 frames per second and include vital data for recovering 3D point clouds from the video sequences. Additionally, pedal signals incorporate information on clutch and camera activations, as well as monopolar and bipolar functionalities crucial for the surgical process. Kinematic data encompasses various transformation matrices and joint states necessary for determining the movement and position of robot arms and the endoscope.

Surgical Task Overview

The dataset outcomes are based on cholecystectomy procedures, which are among the most common laparoscopic interventions. Seven experienced surgeons contributed to this dataset by conducting the procedure three times each on pig livers. The details of the surgical procedures are standardized and the steps followed are consistent with those found in human surgeries, albeit with some adjustments for the experimental nature of the study.

Preliminary Work and Applications

The CRCD dataset enables the development of predictive models for surgeon actions, such as clutch and camera pedal activation, which are essential in streamlining surgical procedures and reducing the surgeons' cognitive load. Through the use of various classifiers, a RandomForest model was able to predict the engagement of the clutch pedal with high accuracy. Furthermore, the dataset empowers the training of more effective segmentation models for tissue recognition and tracking. The ability of these models to adapt to the dynamic changes in tissues during cholecystectomy potentially leads to the automation of some aspects of the surgery, significantly advancing the field of robotic-assisted procedures.