# DexMV: Imitation Learning for Dexterous Manipulation from Human Videos

Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang $^{\boxtimes}$

University of California San Diego, La Jolla 92093, USA

<mailto:xlw012@ucsd.eng.edu>

Abstract. While significant progress has been made on understanding hand-object interactions in computer vision, it is still very challenging for robots to perform complex dexterous manipulation. In this paper, we propose a new platform and pipeline DexMV (Dexterous Manipulation from Videos) for imitation learning. We design a platform with: (i) a simulation system for complex dexterous manipulation tasks with a multi-finger robot hand and (ii) a computer vision system to record large-scale demonstrations of a human hand conducting the same tasks. In our novel pipeline, we extract 3D hand and object poses from videos, and propose a novel demonstration translation method to convert human motion to robot demonstrations. We then apply and benchmark multiple imitation learning algorithms with the demonstrations. We show that the demonstrations can indeed improve robot learning by a large margin and solve the complex tasks which reinforcement learning alone cannot solve. Code and videos are available at <https://yzqin.github.io/dexmv>

Keywords: Dexterous manipulation · Learning from human demonstration · Reinforcement learning

## 1 Introduction

Dexterous manipulation of objects is the primary means for humans to interact with the physical world. Humans perform dexterous manipulation in everyday tasks with diverse objects. To understand these tasks, in computer vision, there is significant progress on 3D hand-object pose estimation [^28] [^78] and affordance reasoning [^13] [^71]. While computer vision techniques have greatly advanced, it is still very challenging to equip robots with human-like dexterity. Recently,

---

there has been a lot of effort on using reinforcement learning (RL) for dexterous manipulation with an anthropomorphic robot hand [^51]. However, given the high Degree-of-Freedom joints and nonlinear tendon-based actuation of the multi-finger robot hand, it requires a large amount of training data with RL. Robot hands trained using only RL will also adopt unnatural behavior. Given these challenges, can we leverage humans' experience in the interaction with the physical world to guide robots, with the help of computer vision techniques?

![Image](images/8bf5d15bf9160ba4ce4be7d0457f137fd86cf6612d19db8ed2c81abb18e6b019.jpg)

<div style="text-align: center;">Fig. 1. We record human videos on manipulation tasks (1st row) and perform 3D hand-object pose estimations from the videos (2nd row) to construct the demonstrations. We have a paired simulation system providing the same dexterous manipulation tasks for the multi-finger robot (3rd row), including relocate, pour, and place inside, which we can solve using imitation learning with the inferred demonstrations.</div>

One promising avenue is imitation learning from human demonstrations [^59] [^64]. Particularly, for dexterous manipulation, Rajeswaran et al. [^59] introduces a simulation environment with four different manipulation tasks and paired sets of human demonstrations collected with a Virtual Reality (VR) headset and a motion capture glove. However, data collection with VR is relatively high-cost and not scalable, and there are only 25 demonstrations collected for each task. It also limits the complexity of the task: It is shown in [^57] that RL can achieve similar performance with or without the demonstrations in most tasks proposed in [^59]. Instead of focusing on a small scale of data, we look into increasing the difficulty and complexity of the manipulation tasks with diverse daily objects. This requires large-scale human demonstrations which are hard to obtain with VR but are much more available from human videos.

In this paper, we propose a new platform and a novel imitation learning pipeline for benchmarking complex and generalizable dexterous manipulation, namely DexMV (Dexterous Manipulation from Videos). We introduce new tasks with the multi-finger robot hand (Adroit Robotic Hand [^41]) on diverse objects in simulation. We collect real human hand videos performing the same tasks as demonstrations. By using human videos instead of VR, it largely reduces the cost for data collection and allows humans to perform more complex and diverse tasks. While the video demonstrations might not be optimal for perfect

---

imitation (e.g., behavior cloning) to learn successful policies, the diverse dataset is beneficial for augmenting the training data for RL, which can learn from both successful and unsuccessful trials.

Our DexMV platform contains a paired systems with: (i) A computer vision system which records the videos of human performing manipulation tasks (1st row in Fig. 1); (ii) A physical simulation system which provides the interactive environments for dexterous manipulation with a multi-finger robot (3rd row in Fig. 1). The two systems are aligned with the same tasks. With this platform, our goal is to bridge 3D vision and robotic dexterous manipulation via a novel imitation learning pipeline.

Our DexMV pipeline contains three stages. First, we extract the 3D hand-object poses from the recorded videos (2nd row in Fig. 1). Unlike previous imitation learning studies with 2-DoF grippers [^69] [^79], we need the human video to guide the 3D-DoF robot hand to move each finger in 3D space. Parsing the 3D structure provides critical and necessary information. Second, a key contribution in our pipeline is a novel demonstration translation method that connects the computer vision system and the simulation system. We propose an optimization-based approach to convert 3D human hand trajectories to robot hand demonstrations. Specifically, the innovations lie in 2 steps: (i) Hand motion retargeting approach to obtain robot hand states; and (ii) Robot action estimation to obtain the actions for learning. Third, given the robot demonstrations, we perform imitation learning in the simulation tasks. We investigate and benchmark algorithms which augment RL objectives with state-only [^57] and state-action [^31] [^59] demonstrations.

We experiment with three types of challenging tasks with the YCB objects [^15]. The first task is to relocate an object to a goal position. Instead of relocating a single ball as [^59], we increase the task difficulty by using diverse objects (first 5 columns in Fig.1). The second task is in the 10th column of the robot to the 1st column of the 1st column of the robot to the 2nd column of the robot to the 2nd column of the robot to the 3rd column of the robot to the 3rd column of the robot to the 4th column of the robot to the 4th column of the robot to the 5th column of the robot to the 5th column of the robot to the 6th column of the robot to the 6th column of the robot to the 7th column of the robot to the 7th column of the robot to the 8th column of the robot to the 8th column of the robot to the 9th column of the robot to the 9th column of the robot to the 10th column of the robot to the 10th column of the robot to the 11th column of the robot to the 11th column of the robot to the 12th column of the robot to the 12th column of the robot to the 13th column of the robot to the 13th column of the robot to the 14th column of the robot to the 14th column of the robot to the 15th column of the robot to the 15th column of the robot to the 16th column of the robot to the 16th column of the robot to the 17th column of the robot to the 17th column of the robot to the 18th column of the robot to the 18th column of the robot to the 19th column of the robot to the 19th column of the robot to the 20th column of the robot to the 20th column of the robot to the 21st column of the robot to the 21st column of the robot to the 22nd column of the robot to the 22nd column of the robot to the 23rd column of the robot to the 23rd column of the robot to the 24th column of the robot to the 24th column of the robot to the 25th column of the robot to the 25th column of the robot to the 26th column of the robot to the 26th column of the robot to the 27th column of the robot to the 27th column of the robot to the 28th column of the robot to the 28th column of the robot to the 29th column of the robot to the 29th column of the robot to the 30th column of the robot to the 30th column of the robot to the 31st column of the robot to the 31st column of the robot to the 32nd column of the robot to the 32nd column of the robot to the 33rd column of the robot to the 33rd column of the robot to the 34th column of the robot to the 34th column of the robot to the 35th column of the robot to the 35th column of the robot to the 36th column of the robot to the 36th column of the robot to the 37th column of the robot to the 37th column of the robot to the 38th column of the robot to the 38th column of the robot to the 39th column of the robot to the 39th column of the robot to the 40th column of the robot to the 40th column of the robot to the 41st column of the robot to the 41st column of the robot to the 42nd column of the robot to the 42nd column of the robot to the 43rd column of the robot to the 43rd column of the robot to the 44th column of the robot to the 44th column of the robot to the 45th column of the robot to the 45th column of the robot to the 46th column of the robot to the 46th column of the robot to the 47th column of the robot to the 47th column of the robot to the 48th column of the robot to the 48th column of the robot to the 49th column of the robot to the 49th column of the robot to the 50th column of the robot to the 50th column of the robot to the 51st column of the robot to the 51st column of the robot to the 52nd column of the robot to the 52nd column of the robot to the 53rd column of the robot to the 53rd column of the robot to the 54th column of the robot to the 54th column of the robot to the 55th column of the robot to the 55th column of the robot to the 56th column of the robot to the 56th column of the robot to the 57th column of the robot to the 57th column of the robot to the 58th column of the robot to the 58th column of the robot to the 59th column of the robot to the 59th column of the robot to the 60th column of the robot to the 60th column of the robot to the 61st column of the robot to the 61st column of the robot to the 62nd column of the robot to the 62nd column of the robot to the 63rd column of the robot to the 63rd column of the robot to the 64th column of the robot to the 64th column of the robot to the 65th column of the robot to the 65th column of the robot to the 66th column of the robot to the 67th column of the robot to the 67th column of the robot to the 68th column of the robot to the 69th column of the robot to the 70th column of the robot to the 70th column of the robot to the 71st column of the robot to the 71st column of the robot to the 72nd column of the robot to the 72nd column of the robot to the 73rd column of the robot to the 73rd column of the robot to the 74th column of the robot to the 74th column of the robot to the 75th column of the robot to the 75th column of the robot to the 76th column of the robot to the 77th column of the robot to the 77th column of the robot to the 78th column of the robot to the 79th column of the robot to the 79th column of the robot to the 80th column of the robot to the 80th column of the robot to the 81st column of the robot to the 81st column of the robot to the 82nd column of the robot to the 82nd column of the robot to the 83rd column of the robot to the 83rd column of the robot to the 84th column of the robot to the 84th column of the robot to the 85th column of the robot to the 85th column of the robot to the 86th column of the robot to the 87th column of the robot to the 87th column of the robot to the 88th column of the robot to the 89th column of the robot to the 90th column of the robot to the 90th column of the robot to the 91st column of the robot to the 91st column of the robot to the 92nd column of the robot to the 92nd column of the robot to the 93rd column of the robot to the 93rd column of the robot to the 94th column of the robot to the 94th column of the robot to the 95th column of the robot to the 95th column of the robot to the 96th column of the robot to the 97th column of the robot to the 97th column of the robot to the 98th column of the robot to the 99th column of the robot to the 100th column of the robot to the 100th column of the robot to the 101st column of the robot to the 101st column of the robot to the 102nd column of the robot to the 102nd column of the robot to the 103rd column of the robot to the 103rd column of the robot to the 104th column of the robot to the 104th column of the robot to the 105th column of the robot to the 105th column of the robot to the 106th column of the robot to the 107th column of the robot to the 107th column of the robot to the 108th column of the robot to the 109th column of the robot to the 110th column of the robot to the 110th column of the robot to the 111st column of the robot to the 111st column of the robot to the 112nd column of the robot to the 112nd column of the robot to the 113rd column of the robot to the 113rd column of the robot to the 114th column of the robot to the 114th column of the robot to the 115th column of the robot to the 115th column of the robot to the 116th column of the robot to the 117th column of the robot to the 117th column of the robot to the 118th column of the robot to the 119th column of the robot to the 120th column of the robot to the 120th column of the robot to the 121st column of the robot to the 121st column of the robot to the 122nd column of the robot to the 122nd column of the robot to the 123rd column of the robot to the 123rd column of the robot to the 124th column of the robot to the 124th column of the robot to the 125th column of the robot to the 125th column of the robot to the 126th column of the robot to the 127th column of the robot to the 127th column of the robot to the 128th column of the robot to the 129th column of the robot to the 130th column of the robot to the 130th column of the robot to the 131st column of the robot to the 131st column of the robot to the 132nd column of the robot to the 132nd column of the robot to the 133rd column of the robot to the 133rd column of the robot to the 134th column of the robot to the 134th column of the robot to the 135th column of the robot to the 135th column of the robot to the 136th column of the robot to the 137th column of the robot to the 137th column of the robot to the 138th column of the robot to the 139th column of the robot to the 140th column of the robot to the 140th column of the robot to the 141st column of the robot to the 141st column of the robot to the 142nd column of the robot to the 142nd column of the robot to the 143rd column of the robot to the 143rd column of the robot to the 144th column of the robot to the 144th column of the robot to the 145th column of the robot to the 145th column of the robot to the 146th column of the robot to the 147th column of the robot to the 147th column of the robot to the 148th column of the robot to the 149th column of the robot to the 150th column of the robot to the 150th column of the robot to the 151st column of the robot to the 151st column of the robot to the 152nd column of the robot to the 152nd column of the robot to the 153rd column of the robot to the 153rd column of the robot to the 154th column of the robot to the 154th column of the robot to the 155th column of the robot to the 155th column of the robot to the 156th column of the robot to the 157th column of the robot to the 157th column of the robot to the 158th column of the robot to the 159th column of the robot to the 160th column of the robot to the 160th column of the robot to the 161st column of the robot to the 161st column of the robot to the 162nd column of the robot to the 162nd column of the robot to the 163rd column of the robot to the 163rd column of the robot to the 164th column of the robot to the 164th column of the robot to the 165th column of the robot to the 165th column of the robot to the 166th column of the robot to the 167th column of the robot to the 167th column of the robot to the 168th column of the robot to the 169th column of the robot to the 170th column of the robot to the 170th column of the robot to the 171st column of the robot to the 171st column of the robot to the 172nd column of the robot to the 172nd column of the robot to the 173rd column of the robot to the 173rd column of the robot to the 174th column of the robot to the 174th column of the robot to the 175th column of the robot to the 175th column of the robot to the 176th column of the robot to the 177th column of the robot to the 177th column of the robot to the 178th column of the robot to the 179th column of the robot to the 180th column of the robot to the 180th column of the robot to the 181st column of the robot to the 181st column of the robot to the 182nd column of the robot to the 182nd column of the robot to the 183rd column of the robot to the 183rd column of the robot to the 184th column of the robot to the 184th column of the robot to the 185th column of the robot to the 185th column of the robot to the 186th column of the robot to the 187th column of the robot to the 187th column of the robot to the 188th column of the robot to the 189th column of the robot to the 190th column of the robot to the 190th column of the robot to the 191st column of the robot to the 191st column of the robot to the 192nd column of the robot to the 192nd column of the robot to the 193rd column of the robot to the 193rd column of the robot to the 194th column of the robot to the 194th column of the robot to the 195th column of the robot to the 195th column of the robot to the 196th column of the robot to the 197th column of the robot to the 197th column of the robot to the 198th column of the robot to the 199th column of the robot to the 200th column of the robot to the 200th column of the robot to the 201st column of the robot to the 201st column of the robot to the 202nd column of the robot to the 202nd column of the robot to the 203rd column of the robot to the 203rd column of the robot to the 204th column of the robot to the 204th column of the robot to the 205th column of the robot to the 205th column of the robot to the 206th column of the robot to the 207th column of the robot to the 207th column of the robot to the 208th column of the robot to the 209th column of the robot to the 210th column of the robot to the 210th column of the robot to the 211st column of the robot to the 211st column of the robot to the 212nd column of the robot to the 213rd column of the robot to the 213rd column of the robot to the 214th column of the robot to the 215th column of the robot to the 215th column of the robot to the 2

- DexMV platform for learning dexterous manipulation using human videos. It contains paired computer vision and simulation systems with multiple dexterous complex manipulation tasks.

- DexMV pipeline to perform imitation learning, with a key innovation on demonstration translation to convert human videos to robot demonstrations.

With DexMV platform and pipeline, we largely improve the dexterous manipulation performance over multiple complex tasks, and its generalization ability to unseen object instances.

---

## 2 Related Work

Dexterous Manipulation. Manipulation with multi-finger hands is one of the most challenging robotics tasks, which has been actively studied with optimization and planning [^3] [^7] [^10] [^20] [^49] [^62]. Recently, researchers have started exploring reinforcement learning for dexterous manipulation [^50] [^51]. However, training with only RL requires huge data samples. Different from a regular 2-DoF robot gripper, the Adroit Robotic Hand used in our experiment has 30 DoFs, which greatly increases the optimization space.

Imitation Learning from Human Demonstrations. Imitation learning is a promising paradigm for robot learning. It is not limited to behavior cloning [^8] [^11] [^54] [^61] [^75] and inverse reinforcement learning [^1] [^5] [^23] [^31] [^44] [^48] [^63] [^76], but also augmenting the RL training with demonstrations [^21] [^53] [^57] [^59] [^77]. For example, Rajeswaran et al. [^59] propose to incorporate demonstrations with on-policy RL. However, all these approaches rely on expert demonstrations collected using expert policies or VR, which is not scalable and generalizable to complex tasks. In fact, it is shown in [^57] that RL can achieve similar performance and sample efficiency with or without the demonstrations in most dexterous manipulation tasks in [^59] besides relocate. Going beyond a VR setup, researchers have recently explored imitation learning and RL with videos [^17] [^64] [^65] [^68] [^69] [^79]. However, all tasks are relatively simple (e.g., pushing a block) with a parallel gripper. In this paper, we propose new challenging dexterous manipulation tasks. We leverage pose estimation for providing demonstrations that largely improve imitation learning performance, while RL alone fails to solve these tasks.

Hand-Object Interaction. Hand-object interaction is a widely studied topic. One line of work focus on object pose estimation [^29] [^32] [^38] [^52] [^56] [^72] [^78] and hand pose estimation [^6] [^12] [^25] [^28] [^33] [^40] [^45] [^70] [^80]. For example, Berk et al. [^15] propose YCB dataset with real objects and their 3D scans. Beyond object poses, datasets and methods for joint estimation of the hand-object poses are also proposed [^18] [^24] [^26] [^45]. Another line of work focus on hand-object contact reasoning [^13] [^71], which can be used for functional grasps [^14] [^34] [^46]. Recently, researchers explored to use hand pose estimation to control a robot hand [^4] [^27] [^43]. They require motion retargeting to map the hand pose into robot motion. These work focus on teleoperation where no objects are involved. In DexMV, we consider the demonstration translation setting, which converts a sequence of hand-object poses into robot demonstrations. We show that the translated demonstration is beneficial for imitation learning for dexterous manipulation.

## 3 Overview

We propose DexMV for Dexterous Manipulation with imitation learning from human Videos including:

---

![Image](images/989df8766e1e2b4713f1df702d070f5d1aaafdc0f2684d045f453d4e5dfff651.jpg)

<div style="text-align: center;">Fig. 2. DexMV platform and pipeline overview. Our platform is composed of a computer vision system (yellow), a simulation system (blue), and a demonstration translation module (colored with green). In computer vision system, we collect human manipulation videos. In simulation, we design the same tasks for the robot hand. We apply 3D hand-object pose estimation from videos, followed by demonstration translation to generate robot demonstrations, which are then used for imitation learning. (Color figure online)</div>

(i) DexMV platform, which offers a paired computer vision system recording human manipulation videos and physical simulators conducting the same tasks. We design three challenging tasks in the simulator and use computer vision system to collect human demonstrations for these tasks in real world. The videos can be efficiently collected with around 100 demonstrations per hour.

(ii) DexMV pipeline, a new pipeline for imitation learning from human videos. Given the recorded videos from the computer vision system, we perform pose estimation and motion retargeting to translate human video to robot hand demonstrations. These demonstrations are then used for policy learning where multiple imitation learning and RL algorithms are investigated. By learning from demonstrations, our policy can be deployed in environments with different goals and object configurations, instead of just following one trajectory.

## 4 DexMV Platform

Our DexMV platform is shown in Fig.2. It is composed of a computer vision system and a simulation system.

Computer Vision System. The computer vision system is used to collect human demonstration videos on manipulating diverse real objects. In this system, we build a cubic frame (35 in. $^{3}$) and attach two RealSense D435 cameras (RGBD cameras) on the top front and top left, as shown on the top row of Fig.2. During data collection, a human will perform manipulation tasks inside the frame, e.g., relocate sugar box. The manipulation videos will be recorded using the two cameras (from a front view and a side view).

---

Simulation System. Our simulation system is built on MuJoCo [^73] with the Adroit Hand [^41]. We design multiple dexterous manipulation tasks aligned with human demonstrations. As shown in the bottom row of Fig. 2, we perform imitation learning by augmenting the Reinforcement Learning with the collected demonstrations. Once the policy is trained, it can be tested on the same tasks with different goals and object configurations.

Task Description. We propose 3 types of manipulation tasks with different objects. We select the YCB objects [^15] with a reasonable size for human to manipulate. For all tasks, the state is composed of robot joint readings, object pose. For relocate, we also include target position. The action is 30-d control command of the position actuator for each joint.

Relocate. It requires the robot hand to pick up an object on the table to a target location. It is inspired by the hardest task in [^59] where the robot hand needs to grasp a sphere and move it to the target position. We further increase the task difficulty by using 5 complex objects as shown in the table above and visualized in the first 5 columns of Fig. 1. The transparent green shape represents the goal, which can change during training and testing (goal-conditioned). The task is successfully solved if the object reaches the goal, without the need for a specific orientation. We train one policy for relocating each object.

Pour. It requires the robot hand to reach the mug and pour the particles inside into a container (Fig.1 column 6). The robot needs to grasp the mug and then manipulate it precisely. The evaluation criteria is based on the percentage of particles poured inside the container.

Place Inside. It requires the robot to pick up an object (e.g., a banana) and place it into a container. The robot needs to rotate the object to a suitable orientation and approach the container carefully to avoid collisions. The evaluation is based on how much percentage of the object mesh volume is inside the container.

Besides testing on the trained objects, we also evaluate the generalization ability of the trained policies on unseen object instances, within the training categories like can, bottle, mug, and also outside the training categories such as camera from ShapeNet dataset [^16].

## 5 Pose Estimation

### 5.1 Object Pose Estimation

We use the 6-DoF pose to represent the location and orientation of objects, which contains translation $T \in \mathbb{R}^3$ and rotation $R \in \mathbf{SO}(3)$. For each frame t in the video, we use the PVN3D [^29] model trained on the YCB dataset [^15] to detect objects and estimate 6-DoF poses. By taking both the RGB image and the point clouds as inputs, the model first estimates the instance segmentation mask. With dense voting on the segmented point clouds, the model then predicts the 3D location of object key points. The 6-DoF object pose is optimized by minimizing the PnP matching error.

---

### 5.2 Hand Pose Estimation

We utilize the MANO model [^60] to represent the hand that consists of hand pose parameter $\theta_t$ for 3D rotations of 15 joints and root global pose $r_t$, and shape parameters $\beta_t$ for each frame t. The 3D hand joints can be computed using hand kinematics function $j_t^{3d} = \mathbf{J}(\theta_t, \beta_t, r_t)$.

Given a video, we use the off-the-shelf skin segmentation [^39] and hand detection [^67] models to obtain a hand mask $M_t$. We use the trained hand pose estimation models in [^45] to predict the 2D hand joints $j_t^{2d}$ and the MANO parameters $\theta_t$ and $\beta_t$ for every frame t using RGB image. We estimate the root joint $r_t$ using the center of the depth image masked by $M_t$. Given the initial estimation $\theta_t$, $\beta_t, r_t$ of each frame t and the camera pose $\Pi$, we formulate the 3D hand joint estimation as an optimization problem,

 $$ \theta_{t}^{*},\beta_{t}^{*},r_{t}^{*}=\underset{\theta_{t},\beta_{t},r_{t}}{\arg\min}||\varPi J(\theta_{t},\beta_{t},r_{t})-j_{t}^{2d}||^{2}+\lambda||M_{t}\cdot(\mathbf{R}(\theta_{t},\beta_{t},r_{t})-D_{t})||^{2} $$

where $\lambda = 0.001$ and $\mathbf{R}$ is a depth rendering function [^37] and $D_t$ is the corresponding depth map in frame t. We minimize the re-projection error in 2D and optimize the hand 3D locations from the depth map. Equation 1 can be further extended to a multi-camera setting by minimizing the objective from different cameras with calibrated extrinsics. We use two cameras by default to handle occlusion. Additionally, we deploy a post-processing procedure with minimizing the difference of $\hat{j}_t^{3d}$ and $\beta_t$ between frames.

![Image](images/b60582e9ae9b4a192fe25e75e1a41fccb323bf6245fc23a84a29d0532d727dd6.jpg)

<div style="text-align: center;">Fig. 3. Top row: Kinematic Chains and Task Space Vectors (TSV). The TSV (dash arrows) are ten vectors to be matched between both hands. Bottom row: (i) Hand motion retargeting; (ii) Action Estimation.</div>

![Image](images/e3f849d023f097930b2ea871e5c06d5fe4d1d018a30d558600c6f82af9166b84.jpg)

<div style="text-align: center;">Fig. 4. Visualization of hand motion retargeting results. Top row: 3D hand-object poses. Mid row: Retargeting results using Finger Tip Mapping as optimization objective. Bottom row: Retargeting results using the proposed optimization objective.</div>

---

## 6 Demonstration Translation

Common imitation learning algorithm consumes state-action pairs from expert demonstrations. It requires the state of the robot and the action of the motor as training data but not directly using the human hand pose. As shown in the kinematics chains in the blue box of Fig. 3, although both robot and human hands share a similar five-finger morphology, their kinematics chains are different. The finger length of each knuckle is also different.

In this paper, we propose a novel method to translate demonstrations from human-centric pose estimation result into robot-centric imitation data. Specifically, there are two steps in demonstration translation (as shown in the red box of Fig.3): (i) Hand motion retargeting to align the human hand motion to robot hand motion, which are with different DoF and geometry; (ii) Predicting robot action, i.e. torque of robot motor: Without any wired sensors, we need to recover the action from only pose estimation results.

### 6.1 Hand Motion Retargeting

In computer graphics and animation, motion retargeting is used to retarget a performer's motion to a virtual character for applications such as movies and cartons [^2] [^30]. While the animation community emphasizes realistic visual appearance, we consider more on the physical effect of the retargeted motion in the context of robotics manipulation. That is, the robot motion should be executable, respecting the physical limit of motors, e.g. acceleration and torque.

Formally, given a sequence of human hand pose estimated $\{(\theta_t, \beta_t, r_t)\}_{t=0}^T$ from a video, the hand pose retargeting can be defined as computing the sequence of robot joint angles $\{q_t\}_{t=0}^T$, where t is the step number in a sequence of length $T + 1$. We formulate the hand pose retargeting as an optimization problem.

Optimization with Task Space Vectors. In most manipulation tasks, human and multi-finger robot hand contacts the object using finger tips. Thus in fingertip-based mapping [^4] [^27], retargeting is solved by preserving the Task Space Vectors (TSV) defined from finger tips position to the hand palm position (green arrow) as shown in the green box of Fig.3, so that the human and robot hands will have same finger tip position relative to the palm. However, only considering finger tip may result in unexpected optimization result as shown in Fig.4. Although the tip position is preserved, the bending information of hand fingers are lost, leading to penetrating the object after retargeting. It becomes more severe when the joint angles are closed to the robot singularity [^47]. To solve this issue, we include the TSV from palm to middle phalanx (blue arrow) as shown in the green box of Fig.3. The optimization objective is,

 $$ \min_{q_{t}}\sum_{i=0}^{N}||\mathbf{v}_{\mathbf{i}}^{\mathbf{H}}(\theta_{t},\beta_{t},r_{t})-\mathbf{v}_{\mathbf{i}}^{\mathbf{R}}(q_{t})||^{2}+\alpha||q_{t}-q_{t-1}||^{2}, $$

where the forward kinematics function for human $\mathbf{v}_{\mathbf{i}}^{\mathbf{H}}(\theta,\beta,r)$ computes the i-th TSV for human hand and the robot forward kinematics function $\mathbf{v}_{\mathbf{i}}^{\mathbf{R}}(q)$ computes

---

i-th TSV for the robot. The first term in Eq. 2 is to find the best $\{q_t\}_{t=0}^T$ that matches the finger tip-palm TSV and finger tip-phalanx TSV from both hands. Note the human hand TSV $\{v_t^H(\theta_0, \beta_0, r_0)\}_{t=0}^T$ is first processed by a low pass filter before optimization for better smoothness. We also add an L2 normalization (second term in Eq. 2) to improve the temporal consistency. For optimization, we use Sequential Least-Squares Quadratic Programming in NLopt [^35], where $\alpha = 8e - 3$ in implementation.

When retargeting hand pose, the objective in Eq.2 is optimized for each t from $t=0$ to $t=T$. For $t\geq1$, we initialize $q_t$ using the optimization results $q_{t-1}$ from last step to further improve the temporal smoothness.

### 6.2 Robot Action Estimation

Hand motion retargeting provides temporal-consistent translation from human hand poses to robot joint angles $\{q_t\}_{t=0}^T$ in different time steps. But the action, i.e. joint torque or control command, is unknown. In robotics, the joint torque $\tau$ can be computed via robot inverse dynamics function $\tau = f_{inv}(q, q', q'')$. To use this function, we first fit the sequence of joint angles into a continuous joint trajectory function $\mathbf{q}(t)$. Then, the first and second order derivative $\mathbf{q}''(t)$ and $\mathbf{q}''(t)$ can be used to compute the torque $\tau(t) = f_{inv}(q(t), q'(t), q''(t))$.

One Important Property of $\mathbf{q}(t)$ is: The absolute value of third-order derivative $\mathbf{q}^{\prime\prime}(t)$, which also refers as jerk, should be as small as possible. We call it minimum-jerk requirement. The reason is two-fold: (i) First, physiological study [^22] shows that the motion of human hand is a minimum jerk trajectory, where the lowest effort is required during the movement. This requirement will resemble the behavior of human, which in turn leads to more natural robot motion. (ii) Second, minimizing jerk can ensure low joint position errors for motors [^42] and limit excessive wear on the physical robot [^19]. Thus we also expect the fitted trajectory function $\{q_t\}_{t=0}^T$ to be minimum-jerk. In implementation, we use the model proposed in [^74] to achieve this.

## 7 Imitation Learning

We perform imitation learning using the translated demonstrations. Instead of using behavior cloning, we adopt imitation learning algorithms that incorporate the demonstrations into RL.

Background. We consider the Markov Decision Process (MDP) represented by  $\langle S, A, P, R, \gamma \rangle$, where  S and  A are state and action space,  $P(s_{t+1}|s_t, a_t)$ is the transition density of state  $s_{t+1}$ at step  $t+1$ given action  $a_t$.  $R(s, a)$ is the reward function, and  $\gamma$ is the discount factor. The goal of RL is to maximize the expected reward under policy  $\pi(a|s)$. We incorporate RL with imitation learning. Given trajectories  $\{(s_i, a_i)\}_{i=1}^n$ from demonstrations  $\pi_D$, we optimize the agent policy  $\pi_\theta$ with  $\{(s_i, a_i)\}_{i=1}^n$ and reward  R. We evaluate imitation learning under two settings: state-action imitation and state-only imitation.

---

### 7.1 State-Action Imitation Learning

We will introduce two algorithms. The first one is the Generative Adversarial Imitation Learning (GAIL) [^31], which is the SOTA IL method that performs occupancy measure matching to learn policy. Occupancy measure $\rho_{\pi}$ [^55] is a density measure of the state-action tuple on policy $\pi$. GAIL aims to minimize the objective $d(\rho_{\pi_{\mathrm{E}}},\rho_{\pi_{\theta}})$, where d is a distance function, $\pi_{\theta}$ is the policy parameterized by $\theta$. GAIL use generative adversarial training to estimate the distance and minimize it with the objective as,

 $$ \begin{array}{r}{\underset{\theta}{\mathrm{m i n}}\underset{w}{\mathrm{m a x}}\underset{s,a\sim\rho_{\pi_{s}}}{\mathbb{D}}\left[\log D_{w}(s,a)\right]+\underset{s,a\sim\rho_{\pi_{\mathrm{D}}}}{\mathbb{E}}\left[\log(1-D_{w}(s,a))\right],}\end{array} $$

where $D_{w}$ is a discriminator. To equip GAIL with reward function, we adopt the approach proposed by [^36], denoted as GAIL+ in this paper.

The second algorithm is Demo Augmented Policy Gradient (DAPG) [^58]. The objective function for policy optimization at each iteration k is as follows.

 $$ g_{a u g}=\sum_{(s,a)\in\rho_{\pi_{\theta}}}\nabla_{\theta}\ln\pi_{\theta}(a|s)A^{\pi_{\theta}}(s,a)+\sum_{(s,a)\in\rho_{\pi_{\mathrm{D}}}}\nabla_{\theta}\ln\pi_{\theta}(a|s)\lambda_{0}\lambda_{1}^{k}, $$

where $A^{\pi_{\theta}}$ is the advantage function [^9] of policy $\pi_{\theta}$, and $\lambda_{0}$ and $\lambda_{1}$ are hyperparameters to determine the advantage of state-action pair in demonstrations.

### 7.2 State-Only Imitation Learning

Besides state-action imitation, we also evaluate the State-Only Imitation Learning (SOIL) [^57] algorithm which does using action information from demonstrations. SOIL extends DAPG to the state-only imitation setting by learning an inverse model $h_{\phi}$ with the collected trajectories when running the policy. The inverse model predicts the missing actions in demonstrations and the policy is trained as DAPG. The policy and inverse model are optimized simultaneously.

## 8 Experiment

Experiment Settings. We adopt TRPO [^66] as our RL baseline. We benchmark imitation learning algorithms SOIL, GAIL+, and DAPG. All these algorithms incorporate demonstrations with TRPO of same hyper-parameters. Thus all the algorithms are comparable. All policies are evaluated with the same three random seeds. By default, the hand pose is estimated using two cameras.

### 8.1 Experiments with Relocate

Main Comparisons. We benchmark four methods: SOIL, GAIL+, DAPG, and RL on the relocate tasks. The success rate is shown in Table 1 and training curves

---

are in Fig.5. A trial is counted as success only when the final position of the object (after 200 steps) is within 0.1 unit length to the target. The initial object position and target position are randomized. In Fig.5, the x-axis is training iterations and the y-axis is normalized average-return over three seeds.

Both Table 1 and Fig. 5 show that the imitation learning methods outperform RL baseline for all five tasks. For mustard bottle and sugar box, a pure RL agent is not able to learn anything. SOIL performs the best for sugar box while DAPG achieves comparable or better performance on mug, mustard bottle, and tomato soup can. Relocate with sugar box is most challenging since it is tall and thin with a flat surface. Once it drops on the table, it is hard to grasp again. We conjecture training the inverse dynamics model online in SOIL helps obtain more accurate action for Relocate.

Ablation on Motion Retargeting Method. Figure 6 (a) illustrates the performance of SOIL with respect to demonstrations generated by different retargeting method, on relocating a tomato soup can. It shows that our retargeting design (blue curve) can improve the imitation learning performance over fingertip mapping (red curve) based retargeting baseline. Besides, without L2 norms in Eq. 2 (green curve), the performance shows a huge drop. It indicates the importance of smooth robot motion when used as demonstrations.

![Image](images/54f75a35c3837874c2ed9c7727d7b11698763e3febd2d0f8a3b421240b579cd1.jpg)

<div style="text-align: center;">Fig. 5. Learning curves of the four methods on the relocate task with respect to five different objects. The x-axis is training iterations. The shaded area indicates standard error and the performance is evaluated with three individual random seeds. (Color figure online)</div>

<div style="text-align: center;">Table 1. Success rate of the evaluated methods on Relocate with five different objects. Success is defined based on the distance between object and target, evaluated via 100 trials for three seeds.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Model</td><td colspan="5">Task - Relocate</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Mustard</td><td style='text-align: center; word-wrap: break-word;'>Sugar Box</td><td style='text-align: center; word-wrap: break-word;'>Tomato Can</td><td style='text-align: center; word-wrap: break-word;'>Clamp</td><td style='text-align: center; word-wrap: break-word;'>Mug</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL</td><td style='text-align: center; word-wrap: break-word;'>$0.33 \pm 0.42$</td><td style='text-align: center; word-wrap: break-word;'>$0.67 \pm 0.47$</td><td style='text-align: center; word-wrap: break-word;'>$0.98 \pm 0.02$</td><td style='text-align: center; word-wrap: break-word;'>$0.89 \pm 0.15$</td><td style='text-align: center; word-wrap: break-word;'>$0.71 \pm 0.35$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GAIL+</td><td style='text-align: center; word-wrap: break-word;'>$0.06 \pm 0.01$</td><td style='text-align: center; word-wrap: break-word;'>$0.00 \pm 0.00$</td><td style='text-align: center; word-wrap: break-word;'>$0.66 \pm 0.47$</td><td style='text-align: center; word-wrap: break-word;'>$0.52 \pm 0.39$</td><td style='text-align: center; word-wrap: break-word;'>$0.53 \pm 0.37$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DAPG</td><td style='text-align: center; word-wrap: break-word;'>$0.93 \pm 0.05$</td><td style='text-align: center; word-wrap: break-word;'>$0.00 \pm 0.00$</td><td style='text-align: center; word-wrap: break-word;'>$1.00 \pm 0.00$</td><td style='text-align: center; word-wrap: break-word;'>$1.00 \pm 0.00$</td><td style='text-align: center; word-wrap: break-word;'>$1.00 \pm 0.00$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RL</td><td style='text-align: center; word-wrap: break-word;'>$0.06 \pm 0.01$</td><td style='text-align: center; word-wrap: break-word;'>$0.00 \pm 0.00$</td><td style='text-align: center; word-wrap: break-word;'>$0.67 \pm 0.47$</td><td style='text-align: center; word-wrap: break-word;'>$0.51 \pm 0.37$</td><td style='text-align: center; word-wrap: break-word;'>$0.49 \pm 0.36$</td></tr></table>

---

Ablation on the Number of Demonstrations. Figure6 (b) shows the performance of SOIL with respect to different number of demonstrations, on relocating a tomato soup can. SOIL can achieve better sample efficiency, performance and the variance is reduced when more demonstrations are given. Table2 illustrates the success rate of the different policies trained with a different number of demonstrations. This shows the importance of our efficient platform to scale up the number and diversity of demonstrations.

Ablation on Environmental Conditions. Figure 6 (c) and (d) illustrates that our demonstrations can transfer to different physical environments. Given the same demonstrations on Relocate the tomato soup can, we train policies with objects scaling into different size from $\times 0.75$ to $\times 1.125$ (Fig. 6 (c)) and applying different frictions (Fig. 6 (d)) from $\times 0.8$ to $\times 1.2$. With same demonstrations, we can still perform imitation learning in different environments and achieve consistent results. All policies achieve much better performance than pure RL. This experiment shows the robustness of our pipeline against the gap between simulation (environment) and real (demonstration).

Ablation on Hand Pose Estimation. We choose 4 different settings for hand pose estimation from video captured by: (i) single camera; (ii) single camera plus post-processing (iii) dual camera; (iv) dual cameras plus the post-processing.

![Image](images/e5b4931af7d1f0cf768bd787de22434b4e1c3f99177fa732011f26ac0a82819c.jpg)

<div style="text-align: center;">(a) Retargeting</div>

![Image](images/9394834af1fe0525d80f21930541e97f22b872a5b7883c082b11b7e0b50dd4e6.jpg)

<div style="text-align: center;">(b) Num of Demo.</div>

![Image](images/0c2d500d2c25376533aec617b983b13ece29da6b4d10e28fb07519c8e21b2977.jpg)

<div style="text-align: center;">(c) Object Size</div>

![Image](images/a1fa5118454ec0f75a7b9e48fd98129aa8b1d66bf9ffb56704d27ff9a0f428df.jpg)

<div style="text-align: center;">(d) Object Friction</div>

<div style="text-align: center;">Fig. 6. Ablation Study: Learning curves of SOIL on Relocate with tomato soup can. The x-axis is training iterations. We ablate: (a) hand pose estimation methods; (b) number of demonstrations used to train SOIL; (c) scaling of object size; (d) friction of the relocated object. The demonstrations are kept the same for all conditions.</div>

<div style="text-align: center;">Table 2. Success rate with different number of demonstrations on Relocate task with a tomato soup can, evaluated via 100 trials for three random seeds at 400, 600, and 800 training iterations.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>#Demo</td><td style='text-align: center; word-wrap: break-word;'>400 Iter.</td><td style='text-align: center; word-wrap: break-word;'>600 Iter.</td><td style='text-align: center; word-wrap: break-word;'>800 Iter.</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL (100)</td><td style='text-align: center; word-wrap: break-word;'>0.36 $\pm$ 0.13</td><td style='text-align: center; word-wrap: break-word;'>0.70 $\pm$ 0.25</td><td style='text-align: center; word-wrap: break-word;'>1.00 $\pm$ 0.00</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL (50)</td><td style='text-align: center; word-wrap: break-word;'>0.19 $\pm$ 0.23</td><td style='text-align: center; word-wrap: break-word;'>0.40 $\pm$ 0.43</td><td style='text-align: center; word-wrap: break-word;'>0.59 $\pm$ 0.39</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL (10)</td><td style='text-align: center; word-wrap: break-word;'>0.04 $\pm$ 0.05</td><td style='text-align: center; word-wrap: break-word;'>0.17 $\pm$ 0.11</td><td style='text-align: center; word-wrap: break-word;'>0.36 $\pm$ 0.21</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RL (0)</td><td style='text-align: center; word-wrap: break-word;'>0.00 $\pm$ 0.00</td><td style='text-align: center; word-wrap: break-word;'>0.00 $\pm$ 0.00</td><td style='text-align: center; word-wrap: break-word;'>0.13 $\pm$ 0.19</td></tr></table>

<div style="text-align: center;">Table 3. Hand pose error and the success rate of learned policy for 4 hand pose estimation settings. Smaller MPJPE means better hand pose estimation performance.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Setting</td><td style='text-align: center; word-wrap: break-word;'>MPJPE</td><td style='text-align: center; word-wrap: break-word;'>Success (%)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>1 Cam</td><td style='text-align: center; word-wrap: break-word;'>41.7</td><td style='text-align: center; word-wrap: break-word;'>66.7 $\pm$ 57.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>1 Cam Post</td><td style='text-align: center; word-wrap: break-word;'>36.2</td><td style='text-align: center; word-wrap: break-word;'>69.7 $\pm$ 33.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2 Cam</td><td style='text-align: center; word-wrap: break-word;'>36.6</td><td style='text-align: center; word-wrap: break-word;'>84.7 $\pm$ 25.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2 Cam Post</td><td style='text-align: center; word-wrap: break-word;'>32.5</td><td style='text-align: center; word-wrap: break-word;'>93.3 $\pm$ 11.5</td></tr></table>

---

![Image](images/63f30aa1d4bbcc36c4592f310e6503252b9f0a8409402772d9cd85f9216dad84.jpg)

<div style="text-align: center;">(a) Pour</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Model</td><td style='text-align: center; word-wrap: break-word;'>Success (%)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL</td><td style='text-align: center; word-wrap: break-word;'>3.5 $\pm$ 3.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GAIL+</td><td style='text-align: center; word-wrap: break-word;'>3.4 $\pm$ 2.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DAPG</td><td style='text-align: center; word-wrap: break-word;'>27.2 $\pm$ 18.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RL</td><td style='text-align: center; word-wrap: break-word;'>1.3 $\pm$ 0.7</td></tr></table>

<div style="text-align: center;">(b) Pour</div>

![Image](images/7db5b79d449dd01feec88d4f5aa216d440653cc35e8cf2248f92ae77df27e236.jpg)

<div style="text-align: center;">(c) Place Inside</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Model</td><td style='text-align: center; word-wrap: break-word;'>Inside Score</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL</td><td style='text-align: center; word-wrap: break-word;'>$27.9 \pm 26.5$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GAIL+</td><td style='text-align: center; word-wrap: break-word;'>$16.0 \pm 14.2$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DAPG</td><td style='text-align: center; word-wrap: break-word;'>$31.3 \pm 30.0$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RL</td><td style='text-align: center; word-wrap: break-word;'>$3.2 \pm 5.6$</td></tr></table>

<div style="text-align: center;">(d) Place Inside</div>

<div style="text-align: center;">Fig. 7. Pour and Place Inside. (a) learning curves of Pour; (b) success rate of Pour; (c) learning rate of Place Inside; (d) Inside score of Place Inside.</div>

(mentioned in Sect. 5.2) Since we have no ground-truth pose annotation in our dataset, we evaluate the performance of hand pose estimation approaches on the DexYCB [^18] dataset, which provides the pose ground-truths. We follow DexYCB to use Mean Per Joint Position Error (MPJPE, smaller the better). We experiment with SOIL on relocating a tomato soup can. The object pose remains the same for all four settings. Table 3 shows better pose estimation in general corresponds to better imitation, except the comparison on 1-Cam Post and 2-Cam. It seems 2-Cam are better than 1-Cam in imitation, even pose estimation result is close. We conjecture 2-Cam can provide more smooth trajectories.

### 8.2 Experiments with Pour

The Pour task involves a sequence of dexterous manipulations: reaching the mug, holding and stably moving the mug, and pouring water into the container. We benchmark four imitation learning algorithms in Fig. 7 (a) and (b). We observe that DAPG converges to a good policy with much fewer iterations: 27.2% of the particles are poured into the container on average. Without the demonstrations, pure RL will only have a very small chance to pour a few particles into the container. State-action method DAPG performs better than state-only method SOIL. It is challenging to learn inverse model with water particles in this task for

![Image](images/5b4fbca364e6e2dfda9fb13aa1875e106a52e8e9f9442116a6f86344ea607b9d.jpg)

<div style="text-align: center;">Setting SOIL GAIL+ DAPG RL</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td colspan="5">Novel Object Instances, Same Category</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>tomato.→can</td><td style='text-align: center; word-wrap: break-word;'>62.0\pm27.3</td><td style='text-align: center; word-wrap: break-word;'>30.7\pm23.5</td><td style='text-align: center; word-wrap: break-word;'>83.6\pm4.7</td><td style='text-align: center; word-wrap: break-word;'>15.9\pm16.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mustard.→bottle</td><td style='text-align: center; word-wrap: break-word;'>0.0\pm0.0</td><td style='text-align: center; word-wrap: break-word;'>0.0\pm0.0</td><td style='text-align: center; word-wrap: break-word;'>68.5\pm7.0</td><td style='text-align: center; word-wrap: break-word;'>0.0\pm0.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mug.→mug</td><td style='text-align: center; word-wrap: break-word;'>51.4\pm33.2</td><td style='text-align: center; word-wrap: break-word;'>33.8\pm36.1</td><td style='text-align: center; word-wrap: break-word;'>79.4\pm11.9</td><td style='text-align: center; word-wrap: break-word;'>27.7\pm38.8</td></tr></table>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>tomato. $\rightarrow$cam.</td><td style='text-align: center; word-wrap: break-word;'>$27.1 \pm 13.5$</td><td style='text-align: center; word-wrap: break-word;'>$5.9 \pm 6.3$</td><td style='text-align: center; word-wrap: break-word;'>$47.2 \pm 2.5$</td><td style='text-align: center; word-wrap: break-word;'>$3.8 \pm 2.1$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mustard. $\rightarrow$cam.</td><td style='text-align: center; word-wrap: break-word;'>$2.6 \pm 2.8$</td><td style='text-align: center; word-wrap: break-word;'>$0.0 \pm 0.0$</td><td style='text-align: center; word-wrap: break-word;'>$49.9 \pm 8.5$</td><td style='text-align: center; word-wrap: break-word;'>$0.1 \pm 0.1$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mug. $\rightarrow$cam.</td><td style='text-align: center; word-wrap: break-word;'>$25.8 \pm 18.8$</td><td style='text-align: center; word-wrap: break-word;'>$17.3 \pm 23.1$</td><td style='text-align: center; word-wrap: break-word;'>$33.4 \pm 3.9$</td><td style='text-align: center; word-wrap: break-word;'>$7.5 \pm 9.5$</td></tr></table>

<div style="text-align: center;">Fig. 8. Relocate generalization. Left: visualization of Relocate with unseen ShapeNet objects from 4 categories. Right: success rate of Relocate. A→B denotes using the policy trained on YCB object A to evaluate the performance on ShapeNet category B.</div>

---

![Image](images/ae4727ced834ab49dc8d928bdd7c43f48799552cd4543d463f586305cfe77eba.jpg)

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Model</td><td style='text-align: center; word-wrap: break-word;'>Success (%)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL</td><td style='text-align: center; word-wrap: break-word;'>3.8 $\pm$ 3.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GAIL+</td><td style='text-align: center; word-wrap: break-word;'>2.1 $\pm$ 0.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DAPG</td><td style='text-align: center; word-wrap: break-word;'>23.3 $\pm$ 10.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RL</td><td style='text-align: center; word-wrap: break-word;'>0.5 $\pm$ 0.3</td></tr></table>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Category</td><td style='text-align: center; word-wrap: break-word;'>Inside Score</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SOIL</td><td style='text-align: center; word-wrap: break-word;'>21.6 $\pm$ 16.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GAIL+</td><td style='text-align: center; word-wrap: break-word;'>10.9 $\pm$ 4.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DAPG</td><td style='text-align: center; word-wrap: break-word;'>26.9 $\pm$ 17.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RL</td><td style='text-align: center; word-wrap: break-word;'>1.8 $\pm$ 1.5</td></tr></table>

<div style="text-align: center;">(a) Visualization of Two Tasks</div>

<div style="text-align: center;">(b) Pour</div>

<div style="text-align: center;">(c) Place Inside</div>

<div style="text-align: center;">Fig. 9. Generalization with Pour and Place Inside. (a) visualization of Pour (top row) with ShapeNet mug, and Place Inside with ShapeNet cellphone; (b) success rate of Pour with ShapeNet Mug; (c) inside score of Place Inside with ShapeNet cellphone.</div>

SOIL, while the analytical inverse dynamics function still provides reasonable actions for DAPG. See our videos and website for visualization.

### 8.3 Experiments with Place Inside

The Place Inside task requires the robot hand to first pick up a banana, rotate it to the appropriate orientation, and place it inside the mug. We define a metric Inside Score, which is computed based on the volume percentage of the banana inside of the mug. We benchmark the imitation learning algorithms in Fig.7 (c) and (d). We find that DAPG outperforms other approaches whereas RL hardly learns to manipulate the object. Between state-only method (SOIL) and state-action method (DAPG), we again observe that computing action offline analytically achieves better results, due to the complexity of the task.

### 8.4 Generalization on Novel Objects and Category

Generalization with Relocate. We benchmark the generalization ability of the trained policy with different imitation learning algorithms. We directly deployed the trained policies for the Relocate task using our demonstrations on: (i) novel unseen objects within the same category; (ii) object instances from a new category. For evaluation on unseen object instances within the same category, we choose the can, bottle, and mug categories from ShapeNet [^16] dataset, which corresponds to the tomato soup can, mustard bottle, and mug in our demonstrations. We omit evaluation on the clamp and sugar box policies since there are no corresponding categories in ShapeNet. For the evaluation on new category, we choose the camera category from ShapeNet. We select 100 object instances for each category and pre-process the objects to fit the size of robot hand. More details about data pre-processing can be found in supplementary. We report the results in Fig. 8. The success rate is computed as a mean accuracy over all object instances within a category. We find DAPG generalizes well on unseen objects in both same category and novel category experiments, significantly outperforming other approaches. We can also observe generalization to novel categories brings a larger challenge.

---

Generalization with Pour. We benchmark the generalization performance of the trained policies from all four methods on new mug instances for Pour task. We choose 100 object instances from the ShapeNet mug category for deployment. The top row of Fig. 9 (a) visualizes the task with different mugs and Fig. 9 (b) shows the success rate. In this benchmark, DAPG still performs the best across all methods. Interestingly, the performance is not much worse than Pour with the trained object (by comparing Fig. 7 (b)). This shows the robustness of imitation learning using our pipeline.

Generalization with Place Inside. For Place Inside, we replace the YCB banana using the objects from cellphone category in ShapeNet as visualized in the bottom row of Fig. 9 (a). Figure 9 (c) shows the generalization performance when directly deploying the policy trained with banana. We find the policy performance is also close to Place Inside with banana (by comparing Fig. 7 (d)). This indicates the robustness of policy against small shape variations and human demonstration on single object can also benefit the tasks on new objects.

## 9 Conclusion

To the best of our knowledge, DexMV is the first work to provide a platform on computer vision/simulation systems and a pipeline on learning dexterous manipulation from human videos. We propose a novel demonstration translation module to bridge the gap between these two systems. We hope DexMV opens new research opportunities for benchmarking imitation learning algorithms for dexterous manipulation.

Acknowledgement. This work was supported, in part, by grants from DARPA LwLL, NSF CCF-2112665 (TILOS), NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 CC*DNI Pacific Research Platform, and gifts from Meta, Google, Qualcomm and Picsart.

## References

[^1]: Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning (2004)

[^2]: Aberman, K., Wu, R., Lischinski, D., Chen, B., Cohen-Or, D.: Learning characteristic motion for motion retargeting in 2d. arXiv preprint arXiv:1905.01680 (2019)

[^3]: Andrews, S., Kry, P.G.: Goal directed multi-finger manipulation: control policies and analysis. Comput. Graph. 37(7), 830–839 (2013)

[^4]: Antotsiou, D., Garcia-Hernando, G., Kim, T.K.: Task-oriented hand motion retargeting for dexterous manipulation imitation. In: ECCV Workshops (2018)

[^5]: Aytar, Y., Pfaff, T., Budden, D., Paine, T., Wang, Z., de Freitas, N.: Playing hard exploration games by watching youtube. In: NeurIPS (2018)

[^6]: Baek, S., Kim, K.I., Kim, T.K.: Pushing the envelope for RGB-based dense 3d hand pose estimation via neural rendering. In: CVPR (2019)

---

[^7]: Bai, Y., Liu, C.K.: Dexterous manipulation using both palm and fingers (2014)

[^8]: Bain, M., Sammut, C.: A framework for behavioural cloning. In: Machine Intelligence (1995)

[^9]: Baird III, L.C.: Advantage updating. Technical Report (1993)

[^10]: Bicchi, A.: Hands for dexterous manipulation and robust grasping: a difficult road toward simplicity. IEEE Trans. Robot. Autom. 16(6), 652–662 (2000)

[^11]: Bojarski, M., et al.: End to end learning for self-driving cars. arXiv (2016)

[^12]: Boukhayma, A., Bem, R.D., Torr, P.H.: 3d hand shape and pose from images in the wild. In: CVPR (2019)

[^13]: Brahmbhatt, S., Ham, C., Kemp, C.C., Hays, J.: Contactdb: analyzing and predicting grasp contact via thermal imaging. In: CVPR (2019)

[^14]: Brahmbhatt, S., Handa, A., Hays, J., Fox, D.: Contactgrasp: functional multi-finger grasp synthesis from contact. arXiv (2019)

[^15]: Calli, B., Walsman, A., Singh, A., Srinivasa, S., Abbeel, P., Dollar, A.M.: Benchmarking in manipulation research: the YDB object and model set and benchmarking protocols. arXiv (2015)

[^16]: Chang, A.X., et al.: Shapenet: an information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)

[^17]: Chang, M., Gupta, A., Gupta, S.: Semantic visual navigation by watching Youtube videos. In: NIPS (2020)

[^18]: Chao, Y.W., et al.: Dexycb: a benchmark for capturing hand grasping of objects. In: CVPR (2021)

[^19]: Craig, J.J.: Introduction to Robotics: Mechanics and Control, 3/E. Pearson Education India, Noida (2009)

[^20]: Dogar, M.R., Srinivasa, S.S.: Push-grasping with dexterous hands: mechanics and a method (2010)

[^21]: Duan, Y., Chen, X., Houthooft, R., Schulman, J., Abbeel, P.: Benchmarking deep reinforcement learning for continuous control (2016)

[^22]: Flash, T., Hogan, N.: The coordination of arm movements: an experimentally confirmed mathematical model. J. Neurosci. 5(7), 1688–1703 (1985)

[^23]: Fu, J., Luo, K., Levine, S.: Learning robust rewards with adversarial inverse reinforcement learning. arXiv (2017)

[^24]: Garcia-Hernando, G., Yuan, S., Baek, S., Kim, T.K.: First-person hand action benchmark with RGB-D videos and 3d hand pose annotations. In: CVPR (2018)

[^25]: Ge, L., et al.: 3d hand shape and pose estimation from a single RGB image. In: CVPR (2019)

[^26]: Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: Honnotate: a method for 3d annotation of hand and object poses. In: CVPR (2020)

[^27]: Handa, A., et al.: Dexpilot: vision-based teleoperation of dexterous robotic hand-arm system. In: ICRA (2020)

[^28]: Hasson, Y., et al.: Learning joint reconstruction of hands and manipulated objects. In: CVPR (2019)

[^29]: He, Y., Sun, W., Huang, H., Liu, J., Fan, H., Sun, J.: Pvn3d: a deep point-wise 3d keypoints voting network for 6dof pose estimation. In: CVPR (2020)

[^30]: Hecker, C., Raabe, B., Enslow, R.W., DeWeese, J., Maynard, J., van Prooijen, K.: Real-time motion retargeting to highly varied user-created morphologies. ACM Trans. Graph. (TOG) 27(3), 1–11 (2008)

[^31]: Ho, J., Ermon, S.: Generative adversarial imitation learning. In: NeurIPS (2016)

[^32]: Hu, Y., Hugonot, J., Fua, P., Salzmann, M.: Segmentation-driven 6d object pose estimation. In: CVPR (2019)

---

[^33]: Iqbal, U., Molchanov, P., Breuel Juergen Gall, T., Kautz, J.: Hand pose estimation via latent 2.5 d heatmap regression. In: ECCV (2018)

[^34]: Jiang, H., Liu, S., Wang, J., Wang, X.: Hand-object contact consistency reasoning for human grasps generation. arXiv (2021)

[^35]: Johnson, S.G.: The nlopt nonlinear-optimization package (2014)

[^36]: Kang, B., Jie, Z., Feng, J.: Policy optimization with demonstrations. In: ICML (2018)

[^37]: Kato, H., Ushiku, Y., Harada, T.: Neural 3d mesh renderer. In: CVPR (2018)

[^38]: Kehl, W., Manhardt, F., Tombari, F., Ilic, S., Navab, N.: Ssd-6d: making RGB-based 3d detection and 6d pose estimation great again. In: ICCV (2017)

[^39]: Khaled, S.M., et al.: Combinatorial color space models for skin detection in sub-continental human images. In: IVIC (2009)

[^40]: Kulon, D., Guler, R.A., Kokkinos, I., Bronstein, M.M., Zafeiriou, S.: Weakly-supervised mesh-convolutional hand reconstruction in the wild. In: CVPR (2020)

[^41]: Kumar, V., Xu, Z., Todorov, E.: Fast, strong and compliant pneumatic actuation for dexterous tendon-driven hands. In: ICRA (2013)

[^42]: Kyriakopoulos, K.J., Saridis, G.N.: Minimum jerk path generation. In: Proceedings. 1988 IEEE International Conference on Robotics and Automation, pp. 364–369. IEEE (1988)

[^43]: Li, S., et al.: Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In: ICRA (2019)

[^44]: Liu, F., Ling, Z., Mu, T., Su, H.: State alignment-based imitation learning. In: ICLR (2020)

[^45]: Liu, S., Jiang, H., Xu, J., Liu, S., Wang, X.: Semi-supervised 3d hand-object poses estimation with interactions in time. In: CVPR (2021)

[^46]: Mandikal, P., Grauman, K.: Dexterous robotic grasping with object-centric visual affordances. arXiv (2020)

[^47]: Nakamura, Y., Hanafusa, H.: Inverse kinematic solutions with singularity robustness for robot manipulator control (1986)

[^48]: Ng, A.Y., Russell, S.J., et al.: Algorithms for inverse reinforcement learning (2000)

[^49]: Okamura, A.M., Smaby, N., Cutkosky, M.R.: An overview of dexterous manipulation. In: ICRA (2000)

[^50]: Akkaya, I., et al.: Solving rubik's cube with a robot hand. OpenAI, arXiv (2019)

[^51]: Andrychowicz, M., et al.: Learning dexterous in-hand manipulation. OpenAI, arXiv (2018)

[^52]: Peng, S., Liu, Y., Huang, Q.X., Bao, H., Zhou, X.: Pvnet: pixel-wise voting network for 6 dof pose estimation. In: CVPR (2019)

[^53]: Peters, J., Schaal, S.: Reinforcement learning of motor skills with policy gradients. Neural Netw. 21(4), 682–697 (2008)

[^54]: Pomerleau, D.A.: Alvinn: an autonomous land vehicle in a neural network. In: NeurIPS (1989)

[^55]: Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (1994)

[^56]: Rad, M., Lepetit, V.: Bb8: a scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth. In: ICCV (2017)

[^57]: Radosavovic, I., Wang, X., Pinto, L., Malik, J.: State-only imitation learning for dexterous manipulation. In: IROS (2021)

[^58]: Rajeswaran, A., et al.: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv (2017)

---

[^59]: Rajeswaran, A., et al.: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations (2018)

[^60]: Romero, J., Tzionas, D., Black, M.J.: Embodied hands: modeling and capturing hands and bodies together. In: ToG (2017)

[^61]: Ross, S., Bagnell, D.: Efficient reductions for imitation learning. In: AISTATS (2010)

[^62]: Rus, D.: In-hand dexterous manipulation of piecewise-smooth 3-d objects. Int. J. Robot. Res. 18(4), 355–381 (1999)

[^63]: Russell, S.: Learning agents for uncertain environments (1998)

[^64]: Schmeckpeper, K., Rybkin, O., Daniilidis, K., Levine, S., Finn, C.: Reinforcement learning with videos: combining offline observations with interaction. arXiv (2020)

[^65]: Schmeckpeper, K., et al.: Learning predictive models from observation and interaction. arXiv (2019)

[^66]: Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: ICML (2015)

[^67]: Shan, D., Geng, J., Shu, M., Fouhey, D.: Understanding human hands in contact at internet scale. In: CVPR (2020)

[^68]: Shao, L., Migimatsu, T., Zhang, Q., Yang, K., Bohg, J.: concept2robot: learning manipulation concepts from instructions and human demonstrations. In: RSS (2020)

[^69]: Song, S., Zeng, A., Lee, J., Funkhouser, T.: Grasping in the wild: learning 6 dof closed-loop grasping from low-cost demonstrations. Robot. Autom. Lett. 5(3), 4978–4985 (2020)

[^70]: Spurr, A., Song, J., Park, S., Hilliges, O.: Cross-modal deep variational hand pose estimation. In: CVPR (2018)

[^71]: Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: Grab: a dataset of whole-body human grasping of objects. In: ECCV (2020)

[^72]: Tekin, B., Sinha, S.N., Fua, P.: Real-time seamless single shot 6d object pose prediction. In: CVPR (2018)

[^73]: Todorov, E., Erez, T., Tassa, Y.: Mujoco: a physics engine for model-based control. In: IROS (2012)

[^74]: Todorov, E., Jordan, M.I.: Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. J. Neurophysiol. 80(2), 696–714 (1998)

[^75]: Torabi, F., Warnell, G., Stone, P.: Behavioral cloning from observation. arXiv (2018)

[^76]: Torabi, F., Warnell, G., Stone, P.: Generative adversarial imitation from observation. arXiv (2018)

[^77]: Večerík, M., et al.: Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv (2017)

[^78]: Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: Posecnn: a convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv (2018)

[^79]: Young, S., Gandhi, D., Tulsiani, S., Gupta, A., Abbeel, P., Pinto, L.: Visual imitation made easy. arXiv (2020)

[^80]: Zimmermann, C., Brox, T.: Learning to estimate 3d hand pose from single RGB images. In: CVPR (2017)