GPT-Fabric: Smoothing and Folding Fabric by Leveraging Pre-Trained Foundation Models

Fabric manipulation has applications in folding blankets, han- dling patient clothing, and protecting items with covers. It is challenging for robots to perform fabric manipulation since fabrics have infinite- dimensional configuration spaces, complex dynamics, and may be in folded or crumpled configurations with severe self-occlusions. Prior work on robotic fabric manipulation relies either on heavily engineered setups or learning-based approaches that create and train on robot-fabric inter- action data. In this paper, we propose GPT-Fabric for the canonical tasks of fabric smoothing and folding, where GPT directly outputs an action informing a robot where to grasp and pull a fabric. We perform extensive experiments in simulation to test GPT-Fabric against prior methods for smoothing and folding. GPT-Fabric matches the state-of-the-art in fabric smoothing, and also achieves comparable performance with most prior fabric folding methods tested, even without explicitly training on a fabric- specific dataset (i.e., zero-shot manipulation). Furthermore, we apply GPT-Fabric in physical experiments over 10 smoothing and 12 folding rollouts. Our results suggest that GPT-Fabric is a promising approach for high-precision fabric manipulation tasks. Code, prompts, videos, and supplementary material are available at https://tinyurl.com/gptfab.

Recommended citation: Vedant Raval*, Enyu Zhao*, Hejia Zhang, Stefanos Nikolaidis, and Daniel Seita. (2024). "GPT-Fabric: Smoothing and Folding Fabric by Leveraging Pre-Trained Foundation Models." International Symposium of Robotics Research (ISRR).
Download Paper