UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding

Abstract: This paper explores the development of UniFolding, a sample-efficient, scalable, and generalizable robotic system for unfolding and folding various garments. UniFolding employs the proposed UFONet neural network to integrate unfolding and folding decisions into a single policy model that is adaptable to different garment types and states. The design of UniFolding is based on a garment's partial point cloud, which aids in generalization and reduces sensitivity to variations in texture and shape. The training pipeline prioritizes low-cost, sample-efficient data collection. Training data is collected via a human-centric process with offline and online stages. The offline stage involves human unfolding and folding actions via Virtual Reality, while the online stage utilizes human-in-the-loop learning to fine-tune the model in a real-world setting. The system is tested on two garment types: long-sleeve and short-sleeve shirts. Performance is evaluated on 20 shirts with significant variations in textures, shapes, and materials. More experiments and videos can be found in the supplementary materials and on the website: https://unifolding.robotflow.ai

• The paper presents a novel UniFolding system that integrates a ResUNet3D and Transformer-based UFONet to automate garment unfolding and folding.
• It employs sparse 3D point cloud data and a dual-phase training pipeline combining VR demonstrations with online self-supervised fine-tuning for enhanced performance.
• Experimental results demonstrate superior sample efficiency and scalability across various garment types, while highlighting challenges with overlapping layers.

Overview of UniFolding: Robotic Garment Folding System

The paper "UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding" (2311.01267) presents a novel robotic system aimed at automating the unfolding and folding of garments. This system, called UniFolding, addresses challenges such as sample efficiency, scalability, and generalization across different garment types and states. UniFolding utilizes a neural network architecture known as UFONet to integrate decisions into a cohesive policy model, leveraging sparse point cloud data to enhance generalization and minimize dependency on texture and shape variations. The paper provides detailed insights into the pipeline of UniFolding, defining action primitives, and training strategies for effective implementation.

System Architecture

UniFolding is structured around two primary components: the unfolding and folding stages, as visualized in the manipulation pipeline.

Figure 1: The manipulation pipeline of UniFolding system. It contains two stages to fully fold a garment from an initial crumpled state, namely Unfolding and Folding.

UFONet Design

UFONet, an end-to-end neural network, serves as the backbone for decision-making within UniFolding. It processes 3D point cloud data from an observed garment state to predict necessary actions. The UFONet architecture employs a ResUNet3D model for feature extraction, followed by a Transformer-based attention mechanism to derive global and dense features. Action classification and regression of action parameters are achieved through dedicated branches within UFONet, allowing for nuanced control over unfolding and folding tasks.

Figure 2: Left: UFONet takes a masked point cloud of the observed garment state as the input, predicts the primitive action type, and regresses the actioning points. Right: The offline and online training strategies for UFONet.

Action Primitives

UniFolding uses a set of predefined action primitives tailored for garment manipulation, namely Fling and Pick-and-Place. The Fling action adapts parameters to accommodate the Flexiv Rizon robot, while Pick-and-Place involves precise placement and release actions to achieve desired garment configurations. These actions ensure adaptability and robustness in manipulating a wide range of garments.

Training Pipeline

UniFolding emphasizes a low-cost, efficient data collection strategy. It incorporates a hybrid training approach that combines human demonstrations in VR with online human-in-the-loop fine-tuning.

Offline and Online Training

Offline training harnesses VR-generated demonstrations to establish initial policy models. The subsequent online stage entails self-supervised refinement in simulation and real-world fine-tuning. This dual-phase strategy significantly enhances the model's adaptability to diverse real-world conditions.

Experimental Evaluation

The effectiveness of UniFolding was validated across multiple garment categories, including long-sleeve and short-sleeve shirts. Performance was assessed in terms of garment unfolding quality and success rates for complete unfolding and folding sequences.

Figure 3: The figure illustrates the shape transformations of different types and sizes of clothes after applying each primitive action of the UniFolding system under various initial states.

Results

UniFolding demonstrated superior sample efficiency and scalability compared to existing methods like ClothFunnels. The incorporation of human annotations in the fine-tuning phase further improved performance, underlining the value of real-world data in bridging sim2real gaps.

Limitations and Future Directions

Despite its strengths, UniFolding faces challenges in handling garments with overlapping layers and self-entanglement scenarios. The authors suggest that future work could expand UniFolding's capabilities to include a broader range of garment categories and enhance its dexterity to address these complex states more effectively.

Conclusion

UniFolding represents a significant advancement in robotic garment manipulation, combining sophisticated neural network models with efficient data-driven training methodologies. Its design and implementation pave the way for further developments in automating intricate tasks like garment folding, pointing towards a future where robotic systems can assist with everyday domestic and industrial tasks with greater autonomy and reliability.