Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing

Recent progress in reinforcement learning (RL) and tactile sensing has significantly advanced dexterous manipulation. However, these methods often utilize simplified tactile signals due to the gap between tactile simulation and the real world.

We present a sensor model for tactile skin that enables zero-shot sim-to-real transfer of ternary shear and binary normal forces. Using this model, we develop an RL policy that leverages sliding contact for dexterous in-hand translation.

We conduct extensive real-world experiments to assess how tactile sensing facilitates policy adaptation to various unseen object properties and robot hand orientations. We demonstrate that our 3-axis tactile policies consistently outperform baselines that use only shear forces, only normal forces, or only proprioception.

To demonstrate the effectiveness of our simulated tactile skin model, we perform sim-to-real experiments on the in-hand translation task. This task requires rich, controlled sliding contact on the palm, so we hypothesize that palm tactile feedback can be particularly useful.

We first test in-domain policy performance on the canonical cylinder object, to compare Proprio-Only and a 3-axis tactile policy. Then, with the same cylinder, we test policy adaptation to OOD hand tilt angles against gravity. These tilt angles passively bias object motion opposing the desired translation direction, thus requiring more force from fingers and gait adaptation to complete the task. This experiment isolates the effect of tilting the hand. Finally, we test policy adaptation to OOD objects and hand tilt angles. We are interested in the limits of policy adaptation, so experiments focus on the maximum angles at which policies are still capable of completing the task.

Notably, we conduct all of our evaluations in the real-world for 190 total rollouts.

A. Real-world cylinder rollouts with S3-Axis and Proprio-Only. This shows superior S3-Axis policy performance compared to Proprio-Only for both ID and OOD conditions. Error bars indicate standard deviation.

B. Three-axis tactile sensing policies demonstrate the best adaptation to OOD objects and unseen hand orientations. S3-axis enables 93% average success rate and +51% increase in distance over Proprio-Only. U3-axis enables +60% increase in velocity over Proprio-Only. These metrics are averaged over all real-world OOD experiments (30 rollouts/policy).

We compare the standard deviation of intersections through Poincare sections of the phase portraits for each finger motor and find that on average, all tactile policies explore more joint states relative to Proprio Only, particularly for the hammer and water bottle experiments.

Greater joint state exploration is potentially a factor in task success and an indicator of gait adaptation; however, other variables such as gait cycle timing and finger coordination are also important to consider.

We demonstrate the generalization of our 3-axis tactile policies to diverse objects, outside of the 4 real world objects in our test set.

All items are being manipulated by either the U3-Axis policy or the S3-Axis policy. The weights of the objects range from 57 g (paper towel roll) to 524 g (rolling pin).

This work was conducted while Jessica Yin was an intern and Haozhi Qi was a research fellow with Meta FAIR. In their academic roles at UC Berkeley, Haozhi Qi and Jitendra Malik are supported in part by ONR MURI N0001421-1-2801. In her academic role at the University of Pennsylvania, Jessica Yin was supported by the NSF Graduate Research Fellowship under Grant No. 202095381.

We thank Mustafa Mukadam, Mike Lambeta, Tingfan Wu, Luis Pineda, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Raunaq Bhirangi, Carolina Higuera, Akash Sharma, Suddhu Suresh, and William Yang for helpful discussions and feedback throughout this work. We thank Dr. Nadia Figueroa, Ho Jin Choi, Tianyu Li, and Harshil Parekh for access and assistance with the Franka robot arm and Optitrack system for hardware experiments.