Grasp contact between hand and object: Capture, analysis, and applications

Abstract

Contact is an important but often oversimplified component of human grasping. Capturing hand-object contact in detail can lead to important insights about grasping behavior, and enable applications in diverse fields like virtual reality and human-robot interaction. However, observing contact through external sensors is challenging because of occlusion and the complexity of the human hand. Lack of ground-truth data has significantly influenced research in this field. This thesis introduces the use of thermal cameras to capture detailed ground-truth hand-object contact (called contact maps), and techniques to simultaneously capture other data modalities like 3D hand pose, object pose, and multi-view RGB-D grasp videos. This has resulted in ContactDB and ContactPose, two large-scale and diverse datasets of participants grasping 3D-printed household objects with functional intents. Analysis of this data confirms some long held intuitions about hand-object contact, and also reveals some surprising new patterns. We also train machine learning models for diverse contact map prediction from object shape, and for contact modeling from object shape and grasp information. Next, this thesis presents ContactGrasp, an algorithm that uses object shape and a contact map to synthesize functional grasps for kinematically diverse hand models, including robotic end-effectors. Finally, this thesis investigates whether the contact data captured by thermal cameras encodes contact pressure in addition to contact locations. We find that (subject to certain conditions) the structure of our contact data indeed includes information about contact pressure.