# Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning

Runyu Ding $^{1,2*}$, Yuzhe Qin $^{1*}$, Jiyue Zhu $^{1*}$, Chengzhe Jia $^{1}$, Shiqi Yang $^{1}$, Ruihan Yang $^{1}$, Xiaojuan Qi $^{2}$ and Xiaolong Wang $^{1}$

Abstract— Teleoperation is a crucial tool for collecting human demonstrations, but controlling robots with bimanual dexterous hands remains a challenge. Existing teleoperation systems struggle to handle the complexity of coordinating two hands for intricate manipulations. We introduce Bunny-VisionPro, a real-time bimanual dexterous teleoperation system that leverages a VR headset. Unlike previous vision-based teleoperation systems, we design novel low-cost devices to provide haptic feedback to the operator, enhancing immersion. Our system prioritizes safety by incorporating collision and singularity avoidance while maintaining real-time performance through innovative designs. Bunny-VisionPro outperforms prior systems on a standard task suite, achieving higher success rates and reduced task completion times. Moreover, the high-quality teleoperation demonstrations improve downstream imitation learning performance, leading to better generalizability. Notably, Bunny-VisionPro enables imitation learning with challenging multistage, long-horizon dexterous manipulation tasks, which have rarely been addressed in previous work. Our system's ability to handle bimanual manipulations while prioritizing safety and real-time performance makes it a powerful tool for advancing dexterous manipulation and imitation learning. Our web page is available at <https://dingry.github.io/projects/bunny_visionpro>.

### I. INTRODUCTION

Playing a Virtual Reality (VR) game is an immersive and intuitive experience where your hand and arm movements seamlessly translate into the action of a virtual character. Now, imagine controlling a bimanual robot in the real world with the same ease: operators use their own movements to guide the robot's motion, just as they would in a VR game. This paradigm shift in robot teleoperation opens up exciting possibilities for more intuitive and accessible human-robot interaction. Recent advancements in VR technology, such as the Apple Vision Pro, have made this concept a possibility.

![Image](images/533695836b761ddd5f1a1b866ff10b075a04f577e252d1726552ab42b08d9028.jpg)

Yet, translating this concept into a practical teleoperation system presents significant challenges due to the intricate motion required to perform human-like manipulations. The complexity is further amplified for high Degree-of-Freedom (DoF) hand-arm systems, where the operator must coordinate two arms and hands to execute tasks requiring spatiotemporal synchronization. Achieving responsive control is crucial, as delays can lead to imprecise robot motion [^1]. Besides, ensuring safety by mitigating risks such as collision and singularity adds another layer of complexity [^2].

In this paper, we introduce Bunny-VisionPro, a novel real-time bimanual dexterous teleoperation system. As illustrated

![Image](images/23ccb41ef48c686ad6ba7d55ccded0ecf88100b2ad0212c8d131d74fe413789c.jpg)

(b) Teleoperation tasks for imitation learning

<div style="text-align: center;">Fig. 1: System Overview and Task Suits. (a) Hand poses captured by Apple Vision Pro are converted into robot motion control commands for real-time teleoperation. The robot provides sensory feedback, including vision and touch, to operators via Vision Pro and actuator-equipped finger cots. (b) We design diverse short-horizon (left-column) and long-horizon tasks (right-column) to evaluate teleoperation performance and its application for imitation learning.</div>

in Fig. 1, our system utilizes Vision Pro's tracking capabilities to capture operator hand movements and translate these movements into precise robotic commands. It addresses both arm and dexterous hand control, prioritizing safety and real-time performance. Crucially, Bunny-VisionPro features three innovative modules: (i) The arm motion control module, which can handle robot singularity and collision avoidance in real time without the need for high-end GPUs. This module ensures the safe and smooth operation of robot arms, even in complex bimanual manipulation scenarios. (ii) The dexterous hand retargeting module, which accurately maps human finger movements to the robotic hand. This module features its unique ability to perform retargeting for robots with loop joints, such as four-bar linkages, in real time. (iii) The haptic feedback module, which uses low-cost Eccentric Rotating Mass (ERM) actuators ($1.2 each) to provide tactile sensations. This novel design, paired with a VR headset, offers a more immersive and realistic experience as the robot interacts with its environment, creating a sensation that the operator is the robot itself.

The advanced capabilities of Bunny-VisionPro enable the collection of high-quality demonstrations for dexterous, bimanual, and long-horizon imitation learning tasks. By leveraging the intuitive and responsive teleoperation system, operators can perform intricate manipulations and collect diverse data for learning algorithms, enabling them to learn and generalize to new scenarios effectively.

We evaluate our Bunny-VisionPro system on the Telekine-

---

sis [^3] benchmark, achieving 11% higher success rates and reducing task completion time by 45% compared to prior systems. These enhancements are especially pronounced in multi-stage, long-horizon tasks. Imitation learning policies trained on demonstrations collected by our system show a 20% improvement in generalization on novel poses and unseen objects compared to using data from previous work [^4]. These results demonstrate the system's superior performance in executing complex bimanual manipulation tasks and its effectiveness in collecting high-quality data.

### II. RELATED WORK

Teleoperation with Gripper. Classical teleoperation methods can be categorized into two main approaches based on their control objectives. The first approach, exemplified by ALOHA [^5], [^6], [^7], uses joint-space mapping [^8] within leader-follower setups. Although enabling impressive bimanual manipulation, this approach is robot-specific, requiring kinematic equivalence between the leader and follower robots [^9]. It also places the burden of managing collision and singularity on the human operator. The second approach prioritizes end-effector control [^10], with arm joint positions calculated using inverse kinematics. Various input devices, such as motion capture systems [^11], [^12], inertia sensors [^13], and VR controllers [^14], [^15], [^16], [^17] have been utilized. Nevertheless, these systems often employ simple 1 or 2 DoF grippers, which limits their dexterity. In contrast, our work aims to teleoperate high-DoF hand-arm systems for complex manipulation tasks.

Dexterous Teleoperation. Teleoperating dexterous hands is challenging due to their high DoF and complex kinematics. Glove-based systems [^18], [^19], [^20], [^21], like the MANUS glove [^22] used by Tesla Bot, can track the operator's finger movements but are costly and require specific hand sizes. Recent vision-based approaches, such as AnyTeleop [^4], facilitate dexterous hand-arm teleoperation using cameras [^3], [^23], [^24], [^25] or VR headsets [^26], [^27], [^28]. However, AnyTeleop requires sophisticated GPU processing for arm motion computation and is primarily designed for a single arm. A concurrent work [^29] controls the Ability hand with reduced DoF using grip buttons, sacrificing dexterous finger-gating to avoid retargeting latency. In contrast, our retargeting modules can handle the Ability hand's four-bar linkage structure at 300Hz with one CPU core, enabling full dexterity. Our system also integrates collision and singularity avoidance while maintaining real-time performance.

Imitation Learning from Demonstration. Imitation learning enables robots to mimic human behaviors through expert guidance. Pioneering research utilizing deep learning [^30], [^31], [^32], [^33] develop policies to generate robot control commands based on image [^5], [^34], [^35], [^36], [^37], [^38] and point cloud data [^39], [^40], [^41], [^42], and further marking progress in bimanual systems [^18], [^5], [^6], [^43]. Additionally, recent works incorporate tactile data [^44], [^29] to enrich the sensory data pool for robot learning. Given that demonstration collection is labor-intensive yet highly critical for effective imitation learning, our system offers a compelling solution that streamlines the collection of high-quality bimanual dexterous data, greatly enhancing accuracy and generalization across various learning algorithms.

### III. TELEOPERATION SYSTEM

### A. Overview

Bunny-VisionPro is a modular teleoperation system using a VR headset's hand and wrist tracking capabilities to control a high-DoF bimanual robot, as shown in Fig. 2. The system consists of three decoupled components: hand motion retargeting, arm motion control, and human haptic feedback.

The hand motion retargeting module maps the operator's finger poses to the robot's dexterous hands, enabling intuitive dexterous manipulation. Concurrently, the arm motion control module uses the operator's wrist poses as input to compute joint angles for the robot arms while considering collision avoidance and singularity handling. The human haptic feedback module converts the tactile readings from sensors mounted on the robot's hands into actuation signals for the wearable Eccentric Rotating Mass (ERM) actuators on the operator's hands. This provides the operator with real-time haptic feedback, enhancing their sense of presence and allowing for more precise manipulation.

Coordinating bimanual robots during teleoperation also requires maintaining the distance between two robot hands to match the distance between two operators' hands, ensuring natural, coordinated movements. However, the initial distance between the robot hands may not match that of the human hands. To address this, we design several initialization modes that align the robot's hands with the operator's hands on the fly when the robot starts to move. Different tasks may benefit from different initialization modes. This step creates a consistent starting point, enabling intuitive and efficient bimanual task execution throughout the teleoperation process.

For communication, we stream hand pose results from Vision Pro to the computer via [^45]. The modular architecture allows for better extensibility and enables each module to run in a separate computation process, preventing latency accumulation in the system and ensuring real-time control.

### B. Robot Hand Motion Retargeting

The hand motion retargeting module translates human finger poses into corresponding robot joint positions. This is achieved by minimizing the difference between the fingertip keypoint vectors [^46] of the human hand and the robotic hand, formulated as an online optimization problem:

 $$ L_{\mathrm{h a n d}}=\sum_{i=1}^{N}||\alpha v_{i}-\mathrm{F K}_{i}(q)||_{2}^{2}+\beta||\Delta q||_{2}^{2}\quad\mathrm{s.t.}\quad q^{l}\leq q\leq q^{u}. $$

The objective function  $L_{\text{hand}}$ consists of two terms: (i) The first term minimizes the distance between the scaled human hand keypoint vectors  $\alpha v_i$ and the robot hand keypoint vectors computed by forward kinematics function  $\text{FK}_i(q)$, where  $v_i$ represents the  $i^{th}$ fingertip keypoint vector,  N is the total number of vectors,  $\|\cdot\|_2$ denotes Euclidean norm,  $\alpha$ is the scaling factor to adjust for size differences between the human and robot hand and  q is the joint positions of the robot hand bounded by the lower and upper limit,  $q^l$ and  $q^u$. (ii)

---

![Image](images/c098fe8822a3731e0dd9381df67d3c11632dc8597362dafc5db4b71419b1d1be.jpg)

<div style="text-align: center;">Fig. 2: Teleoperation System. The operator controls the robot hand and arm using finger and wrist poses, respectively. The system's visual and haptic feedback, combined with its four real-time capabilities, provides an intuitive and immersive VR experience for the operator.</div>

The second term enforces temporal smoothness by penalizing large joint position changes $\Delta q$ between consecutive frames, with weight $\beta$. The optimization variable is q and the objective is solved using Sequential Quadratic Programming (SQP) [^47], [^48], a gradient-based method that iteratively solves quadratic approximations of the problem.

Our work uniquely addresses the real-time handling of loop joints, common in dexterous hands, where the number of joints exceeds the actual DoF. For a robot with n joints, the positions of $n-k$ passive joints depend on the other k active joint, represented as:

 $$ q_{j}=c_{j}(q_{1},q_{2},...,q_{k}),\mathrm{f o r}j\in\{k+1,\ldots,n\}, $$

where the passive joint j's position $q_{j}$ is not independent but constrained by function $c_{j}$. n is the total number of joint positions. Instead of adding equality constraints to the optimization problem in Eq. (1), which can lead to instability and increased computation time due to the highly nonlinear forward kinematic function $\mathrm{FK}(q)$, we reformulate the problem into a reduced dimension where the optimization variable has k dimensions (active joints) instead of n (total joints). Specifically, in the forward pass of optimization, the positions of passive joints are computed as Eq. (2). In the backward pass, the gradients of the passive joints are backpropagated to the active joints and added to their original gradients. Thereby, the gradient for the active joint i is:

 $$ grad_{i}=\frac{\partial L_{\mathrm{hand}}}{\partial q_{i}}+\sum_{j=k+1}^{n}\frac{\partial c_{j}}{\partial q_{i}}\frac{\partial L_{\mathrm{hand}}}{\partial q_{j}}. $$

This approach eliminates the need for SQP to handle affine approximations of the constraints, making the problem more tractable. Experimental results in Tab. I show a $10.2x$ speedup when using this method to solve the four-bar linkage structure (a type of loop joint) in the Ability Hand.

### C. Robot Arm Motion Control

Arm motion control involves computing the joint trajectories of the robot arm based on the wrist pose detected by Vision Pro. Traditional approaches often rely on closed-form solutions for Inverse Kinematics (IK) and handle constraints such as singularity by exploring the null space [^49]. However, for typical 7-DoF arms, the 1-DoF null space may not provide enough flexibility. Modern approaches formulate the problem as an optimization task [^50], similar to retargeting. It relaxes the rigid constraints of IK to accommodate additional factors. Inspired by these works, we have developed a unified and efficient optimization objective $L_{arm}$ in Eq.(4) that integrates IK, collision avoidance, and singularity management for real-time motion control.

 $$ L_{\mathrm{a r m}}=L_{\mathrm{i k}}+L_{\mathrm{s i n}}+L_{\mathrm{c o l}}, $$

The first term concerns the IK objective:

 $$ \begin{array}{r}{L_{\mathrm{i k}}=\beta_{\mathrm{p o s}}||p_{\mathrm{e e}}-p_{\mathrm{w r i s t}}||_{2}+\beta_{\mathrm{r o t}}\cdot\mathrm{a r c c o s}(2\langle q_{\mathrm{e e}},q_{\mathrm{w r i s t}}\rangle^{2}-1).}\end{array} $$

Here, $p_{ee}$ and $p_{wrist}$ represent the robot end effector and human wrist position, respectively, while $q_{ee}$ and $q_{wrist}$ are their corresponding quaternions. $\langle\cdot,\cdot\rangle$ means inner product. The weights $\beta_{pos}$ and $\beta_{rot}$ allow for balancing the errors between position and rotation. Additionally, the objectives for singularity and collision avoidance are as follows:

 $$ L_{\mathrm{s i n}}=\left\{\begin{array}{c c}{1-\frac{\sqrt{\operatorname*{d e t}\left[\mathbf{J J}^{\mathbf{T}}\right]}}{\lambda},}&{\mathrm{i f}s_{0}<s_{\mathrm{l o w}}}\\ {0,}&{\mathrm{o t h e r w i s e}}\\ \end{array}\right., $$

 $$ \begin{array}{r}{L_{\mathrm{c o l}}=\frac{1}{\sum_{i=1}^{m}\sum_{j=1}^{m}\mathsf{d i s t}(e_{i},e_{j})\mathbb{1}(e_{i},e_{j})+\epsilon}.}\end{array} $$

In the singularity avoidance objective $L_{\mathrm{sin}}$, J represents the spatial Jacobian matrix, $s_0$ is the smallest singular value, $s_{\mathrm{low}}$ is a conditional trigger for penalizing singularity, and $\lambda$ acts as a temperature factor. Theoretically, both $s_0$ and the manipulability index $\sqrt{\det[J\mathbf{J}^{\mathrm{T}}]}$ can measure singularity [^51], and the robot is nearing singularity when either value is low. We opt for the manipulability index as the objective due to its suitability for gradient-based optimization while using $s_0$ as the condition. Thus, $L_{\mathrm{sin}}$ only takes effect when the robot approaches singularity.

For collision detection, the GJK algorithm [^52] is often employed but can be time-intensive even with convex meshes. To efficiently compute the self-collision cost $L_{col}$, we model each robot link as a collection of m spheres. The function $\text{dist}(\cdot,\cdot)$ calculates the Euclidean distance between spheres $e_i$ and $e_j$, while $\mathbf{1}(\cdot,\cdot)$ serves as an indicator function to evaluate the necessity of collision checking, with $\epsilon$ provides numerical stability. Collisions are disregarded

---

![Image](images/72e9a754857ab9bdae600d70399fcd23ef0fe71fcf7221866286f55a49ee73fe.jpg)

<div style="text-align: center;">(a) Human haptic feedback from robot tactile readings</div>

![Image](images/aa17444c3975f858169d1fdc4001e5090d4b55f9cd445b9ea60aa31a89563f01.jpg)

<div style="text-align: center;">(b) Zero-drift calibration of tactile data</div>

<div style="text-align: center;">Fig. 3: Human Haptics Feedback Device. Tactile readings from FSR sensors located in the robot's fingertips undergo calibration and low-pass filtering. The processed tactile signals then drive ERM motors to deliver touch feedback to human.</div>

between spheres within the same link or between spheres of a parent and its direct child. This simplification not only accelerates computation but also renders the distance function differentiable.

### D. Human Haptic Feedback Device

Effective human manipulation relies on the integration of both visual and tactile feedback. However, many vision-based teleoperation systems [^4], [^23], [^26], [^28], [^3] neglect haptic feedback. To address this limitation, we have developed a cost-effective haptic feedback system using Eccentric Rotating Mass (ERM) actuators (Fig. 3(a)). This system first processes tactile signals from robot hands (Step I) and then drives vibration motors to simulate tactile sensations (Step II). Despite its low cost, our system enables the operator to perceive and respond to the environment more intuitively with a more immersive experience, leading to improved manipulation performance.

Step I: Tactile Signal Processing. The Ability hand (Fig.3(a)) uses Force-Sensitive Resistors (FSRs) to measure finger pressure. However, FSR sensors suffer from imprecision and zero-drift problems [^53], [^54], which are aggravated by deformable wrapping materials. To address this, we perform zero-drift calibration by recording baseline FSR readings at various joint positions and subtracting interpolated baseline values from real-time readings during operation. This efficient calibration can be performed autonomously each time teleoperation is initiated. Furthermore, we apply a low-pass filter to reduce noise and smooth the tactile data. As shown in Fig. 3(b), these signal-processing techniques significantly improve the quality and reliability of the tactile feedback.

Step II: Vibration Motor Driving. In this step, we convert the processed tactile signals to ERM actuator vibrations using an ELEGOO UNO board. Since ERM motors require constant input voltage, we employ Pulse-Width Modulation (PWM) to simulate continuous haptic strength by controlling ERM vibration intensity through modulated pulse width. To further enhance the stability of the haptic signals and the system's robustness, we add a Bipolar Junction Transistor (BJT) between each ERM actuator and its corresponding

<div style="text-align: center;">TABLE I: Profiling Results. We measure module latencies in our system.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Module</td><td style='text-align: center; word-wrap: break-word;'>Time (ms)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Retargeting:</td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o loop joints</td><td style='text-align: center; word-wrap: break-word;'>2.54</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Loop joints (baseline, as constraints)</td><td style='text-align: center; word-wrap: break-word;'>34.98</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Loop joints (ours)</td><td style='text-align: center; word-wrap: break-word;'>3.43</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Motion Control:</td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>IK only</td><td style='text-align: center; word-wrap: break-word;'>0.74</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Collision</td><td style='text-align: center; word-wrap: break-word;'>7.85</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Singularity</td><td style='text-align: center; word-wrap: break-word;'>10.42</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Collision + Singularity</td><td style='text-align: center; word-wrap: break-word;'>15.93</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Haptic Feedback (PWM)</td><td style='text-align: center; word-wrap: break-word;'>0.04</td></tr></table>

<div style="text-align: center;">TABLE II: Teleoperation Results in Telekinesis Benchmark. 10 single-arm tasks are evaluated.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Task</td><td style='text-align: center; word-wrap: break-word;'>Telekinesis [^3]</td><td style='text-align: center; word-wrap: break-word;'>AnyTeleop [^4]</td><td style='text-align: center; word-wrap: break-word;'>Ours</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pickup Box Object</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>20/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pickup Fabric Toy</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>20/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Box Rotation</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>18/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Scissor Pickup</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>15/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Cup Stack</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>20/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Two Cup Stacking</td><td style='text-align: center; word-wrap: break-word;'>3/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>19/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pouring Cubes onto Plate</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>20/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Cup Into Plate</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>20/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Open Drawer</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>20/20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Open Drawer &amp; Pickup Object</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>20/20</td></tr></table>

PWM pin. This ensures that the haptic feedback of each ERM motor is individually adjusted, only activated during the pulse width, and resistant to circuit noise.

### IV. SYSTEM EVALUATION

We validate our system through: (1) Profiling analysis to verify real-time capabilities (Sec. IV-A); (2) Performance comparison with advanced teleoperation systems on real-robot dexterous tasks (Sec. IV-B); (3) User study with untrained operators to assess haptic usability (Sec. IV-C).

### A. Profiling Results

We profile module latencies in our system using an i7-12700KF CPU (see Tab. I). Our method retargets hand motion on the Ability hand, processing loop joints as efficiently as methods without loop joints and much faster than constraint-based approaches (baseline in Tab. I). Our

---

<div style="text-align: center;">TABLE III: Teleoperation System Evaluation. We measure collection success rate (Success), episode length (EpLen), collection time in minutes (Time), and joint position changes of the arms ( $\Delta Q_{pos}$).</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Task</td><td style='text-align: center; word-wrap: break-word;'>Description</td><td style='text-align: center; word-wrap: break-word;'>Arm</td><td style='text-align: center; word-wrap: break-word;'>System</td><td style='text-align: center; word-wrap: break-word;'>Success ( $\uparrow$)</td><td style='text-align: center; word-wrap: break-word;'>Time ( $\downarrow$)</td><td style='text-align: center; word-wrap: break-word;'>EpLen ( $\downarrow$)</td><td style='text-align: center; word-wrap: break-word;'>$\Delta$Qpos ( $\downarrow$)</td></tr><tr><td colspan="8">Short-horizon Tasks</td></tr><tr><td rowspan="2">Grasp Toy</td><td rowspan="2">Grasp toy and place in box.</td><td rowspan="2">single</td><td rowspan="2">AnyTeleop+ Ours</td><td style='text-align: center; word-wrap: break-word;'>50 / 54</td><td style='text-align: center; word-wrap: break-word;'>28.9</td><td style='text-align: center; word-wrap: break-word;'>145 $\pm$24</td><td style='text-align: center; word-wrap: break-word;'>0.77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>50 / 55</td><td style='text-align: center; word-wrap: break-word;'>24.3</td><td style='text-align: center; word-wrap: break-word;'>142 $\pm$21</td><td style='text-align: center; word-wrap: break-word;'>0.35</td></tr><tr><td rowspan="2">Clean Pan</td><td rowspan="2">Grasp panhandle and sponge to clean the pan.</td><td rowspan="2">dual</td><td rowspan="2">AnyTeleop+ Ours</td><td style='text-align: center; word-wrap: break-word;'>50 / 78</td><td style='text-align: center; word-wrap: break-word;'>74.6</td><td style='text-align: center; word-wrap: break-word;'>282 $\pm$23</td><td style='text-align: center; word-wrap: break-word;'>0.98</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>50 / 58</td><td style='text-align: center; word-wrap: break-word;'>30.4</td><td style='text-align: center; word-wrap: break-word;'>186 $\pm$17</td><td style='text-align: center; word-wrap: break-word;'>0.58</td></tr><tr><td rowspan="2">Uncover &amp; Pour</td><td rowspan="2">Grasp container, remove lid, and pour into bowl.</td><td rowspan="2">dual</td><td rowspan="2">AnyTeleop+ Ours</td><td style='text-align: center; word-wrap: break-word;'>50 / 77</td><td style='text-align: center; word-wrap: break-word;'>63.2</td><td style='text-align: center; word-wrap: break-word;'>189 $\pm$21</td><td style='text-align: center; word-wrap: break-word;'>1.26</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>50 / 63</td><td style='text-align: center; word-wrap: break-word;'>41.5</td><td style='text-align: center; word-wrap: break-word;'>183 $\pm$15</td><td style='text-align: center; word-wrap: break-word;'>0.86</td></tr><tr><td colspan="8">Long-horizon Tasks</td></tr><tr><td rowspan="2">Wipe Glass</td><td rowspan="2">Pull out wipes to clean glass</td><td rowspan="2">dual</td><td rowspan="2">AnyTeleop+ Ours</td><td style='text-align: center; word-wrap: break-word;'>30 / 53</td><td style='text-align: center; word-wrap: break-word;'>69.0</td><td style='text-align: center; word-wrap: break-word;'>517 $\pm$51</td><td style='text-align: center; word-wrap: break-word;'>1.44</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>30 / 41</td><td style='text-align: center; word-wrap: break-word;'>32.2</td><td style='text-align: center; word-wrap: break-word;'>360 $\pm$35</td><td style='text-align: center; word-wrap: break-word;'>0.68</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Sweep Floor</td><td style='text-align: center; word-wrap: break-word;'>Sweep and dump trash using dustpan and broom</td><td style='text-align: center; word-wrap: break-word;'>dual</td><td style='text-align: center; word-wrap: break-word;'>AnyTeleop+ Ours</td><td style='text-align: center; word-wrap: break-word;'>30 / 61</td><td style='text-align: center; word-wrap: break-word;'>129.4</td><td style='text-align: center; word-wrap: break-word;'>786 $\pm$118</td><td style='text-align: center; word-wrap: break-word;'>1.99</td></tr><tr><td rowspan="3">Prepare Coffee</td><td rowspan="3">Pour coffee into bottle, stir and hand it over to human</td><td rowspan="3">dual</td><td rowspan="3">AnyTeleop+ Ours</td><td style='text-align: center; word-wrap: break-word;'>30 / 40</td><td style='text-align: center; word-wrap: break-word;'>45.4</td><td style='text-align: center; word-wrap: break-word;'>570 $\pm$77</td><td style='text-align: center; word-wrap: break-word;'>1.30</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>30 / 52</td><td style='text-align: center; word-wrap: break-word;'>95.3</td><td style='text-align: center; word-wrap: break-word;'>705 $\pm$42</td><td style='text-align: center; word-wrap: break-word;'>1.87</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>30 / 44</td><td style='text-align: center; word-wrap: break-word;'>55.2</td><td style='text-align: center; word-wrap: break-word;'>576 $\pm$33</td><td style='text-align: center; word-wrap: break-word;'>1.05</td></tr></table>

motion control module, including singularity and collision avoidance, operates in real-time at > 60 Hz.

### B. Real Robot Teleoperation Experiments

Task Suits. The Teleskinesis [^3] benchmark comprises ten single-arm-hand manipulation tasks, as listed in Tab. II. To further gauge our system's capability primarily for bimanual dexterous manipulation, we design three short-horizon tasks – grasping toy, cleaning pan and uncovering and pouring – and three long-horizon tasks – wiping glass, sweeping floor and preparing coffee, as shown in Fig. 1. These tasks, except for grasping toy, necessitate fine-grained coordination between two hands.

Real-world Robot Setup. Our bimanual dexterous system consists of two UFactory xArm-7 arms, each equipped with a 6-DoF Ability hand, resulting in a 26-DoF system. Each hand incorporates 30 tactile sensors distributed across five fingertips. For data collection, two RealSense L515 cameras are positioned at the front and the top of the robot's workspace to capture adequate visual observations.

Baselines. We assess our system on the Teleskinesis benchmark against Teleskinesis [^3] and AnyTeleop [^4], using the results reported in the original papers for comparison. For our custom tasks, to ensure a fair comparison, we integrate AnyTeleop with our Vision-Pro-based hand tracking to create AnyTeleop+ and use an identical real-robot setup as our Bunny-VisionPro.

Teleoperation Results. As shown in Tab. II, Bunny-VisionPro matches or surpasses baseline methods in 9 out of 10 tasks, evidencing its system precision and robustness for hand manipulation teleoperation. However, in the scissor pickup task, our system underperforms slightly due to the limited DoF in the Ability hand, which makes it challenging to insert fingers into the scissor handles.

In self-designed manipulation tasks (Tab. III), our system outperforms AnyTeleop+ [^4], achieving 11% higher success rate with 45% task completion time. This performance confirms its precision in replicating human actions and its responsiveness and efficiency. Additionally, BunnyVisionPro demonstrates enhanced stability and robustness, reducing episode lengths by 19% with lower variability. In contrast, AnyTeleop+ struggles with complex trajectories and large end-effector pose changes, often leading to aggressive joint movements and unpredictable, potentially unsafe robot control, as evidenced by a 43% increase in arm joint position changes during tasks. Besides, our system's advantages are particularly pronounced in bimanual long-horizon tasks, where it excels in real-time, fine-grained coordination between two hands, leading to a better teleoperation experience.

### C. User Study of Haptic Feedback

![Image](images/d9ff7384e468e7b9395e63558c40f2bdbe38cdb8c1b5523c55ae002d7eee14c5.jpg)

![Image](images/632b183be6bdbbf442d75ceb52e622a0b154c7bad584d0186940b2bd79228087.jpg)

![Image](images/aaabc60f9b9c0a9b48c8894acfa6cd8f2d0e1e30a83f57381bc01c683e63f0e0.jpg)

![Image](images/527883a73e21007a928ad0d0ee592db981180bde90881a4cdb8096790f5a69e3.jpg)

<div style="text-align: center;">Fig. 4: User Study. It evaluates the impact of haptic feedback on success rates and time efficiency.</div>

We conduct a user study on human haptic feedback with five untrained operators. Participants perform toy handover and ball moving tasks, five trials for each task, with task success rates and average completion time recorded. As shown in Fig. 4, haptic feedback greatly improves success rate and efficiency in the ball-moving task where vision is partially occluded and touch sensing is crucial. This boost stems from the enhanced interaction awareness between the robot and the object, which prevents insufficient or excessive pressure, thus promoting safer and quicker operations. The toy-handover task also sees slight improvements in success rate. This may be attributed to the enhanced confidence in teleoperation reported by operators when using haptic feedback, regardless of visual obstacles. More haptics comparisons are on the project page.

---

<div style="text-align: center;">TABLE IV: Imitation Learning for Real-World Tasks. The Success Rate (SR) is based on 10 trials, with SR-SG indicating success rates for spatial generalization and SR-U for unseen objects.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Task</td><td rowspan="2">Data Collection System</td><td colspan="3">ACT [^5]</td><td colspan="3">Diffusion Policy [^34]</td><td colspan="3">DP3 [^40]</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SR</td><td style='text-align: center; word-wrap: break-word;'>SR-SG</td><td style='text-align: center; word-wrap: break-word;'>SR-U</td><td style='text-align: center; word-wrap: break-word;'>SR</td><td style='text-align: center; word-wrap: break-word;'>SR-SG</td><td style='text-align: center; word-wrap: break-word;'>SR-U</td><td style='text-align: center; word-wrap: break-word;'>SR</td><td style='text-align: center; word-wrap: break-word;'>SR-SG</td><td style='text-align: center; word-wrap: break-word;'>SR-U</td></tr><tr><td rowspan="2">Grasp toy</td><td style='text-align: center; word-wrap: break-word;'>AnyTeleop+</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>3/10</td><td style='text-align: center; word-wrap: break-word;'>3/10</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td></tr><tr><td rowspan="2">Clean pan</td><td style='text-align: center; word-wrap: break-word;'>AnyTeleop+</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>3/10</td><td style='text-align: center; word-wrap: break-word;'>2/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>1/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>1/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td></tr><tr><td rowspan="2">Uncover &amp; pour</td><td style='text-align: center; word-wrap: break-word;'>AnyTeleop+</td><td style='text-align: center; word-wrap: break-word;'>2/10</td><td style='text-align: center; word-wrap: break-word;'>0/10</td><td style='text-align: center; word-wrap: break-word;'>1/10</td><td style='text-align: center; word-wrap: break-word;'>3/10</td><td style='text-align: center; word-wrap: break-word;'>2/10</td><td style='text-align: center; word-wrap: break-word;'>2/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>3/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td><td style='text-align: center; word-wrap: break-word;'>2/10</td><td style='text-align: center; word-wrap: break-word;'>3/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>5/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td></tr></table>

### V. IMITATION LEARNING

To assess the quality of demonstrations collected by our system compared to AnyTeleop+, we evaluate their impact on imitation learning. We train policies using three methods: ACT [^5], Diffusion Policy [^34], and DP3 [^40], then test their generalization to unseen scenarios. This allows us to measure data quality independently of specific learning algorithms. Besides, we investigate the effectiveness of tactile data in imitation learning.

### A. Main Results

As shown in Tab. IV, Bunny-VisionPro significantly outperforms AnyTeleop+ by an average of 22% success rate across three tasks using three different policies. This showcases the superior quality of the demonstrations collected by our system. Our system's effectiveness is particularly evident in bimanual tasks, i.e. cleaning pan and uncovering and pouring. One possible reason for this improvement is that our system handles the alignment of bimanual robots delicately, enabling the collection of demonstrations with more natural robot motions.

Generalization. We examine data quality impact on policy generalization by comparing policies trained on AnyTeleop+’s and Bunny-VisionPro’s data. We assess spatial generalization (SR-SG) with random object poses and manipulation of unseen objects (SR-U) with varying shapes, sizes, and colors. Across three policies trained on each dataset, those using Bunny-VisionPro’s data show consistent gains of 14% SR-SG and 26% SR-U (Tab. IV). It suggests our system enables high-quality, trajectory-stable data collection, which benefits imitation generalization.

Long-horizon Tasks. For complicated, multi-stage tasks, the policy achieves notable success rates – 73% for sub-tasks and 38% for entire tasks – with only 30 demonstrations. This manifests the reliability of long-horizon data collection of our system to serve advanced manipulation task learning.

### B. Tactile Data as Policy Input

Tactile Data Processing. FSR sensors capture the pressure on the robot during object gripping. These signals can also be processed to calculate impulse by differentiating force over time. For tactile learning, touch signals are represented as vectors, encoded using MLPs, or visualized as point sets for visual embedding learning, akin to DexPoint [^55].

Results and Discussion. As shown in Tab. VI, utilizing tactile data does not necessarily enhance outcomes, potentially because vision alone suffice for these tasks. Tactile

<div style="text-align: center;">TABLE V: DP3 Results for Long-Horizon Tasks. SR-subtask is subtask success rate.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Task</td><td colspan="2">DP3 [^40]</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SR-subtask</td><td style='text-align: center; word-wrap: break-word;'>SR</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Wipe glass</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Sweep floor</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Prepare coffee</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td></tr></table>

<div style="text-align: center;">TABLE VI: Ablation of Touch as DP3 Policy Input. Success rates are reported for two bimanual tasks.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Signal</td><td style='text-align: center; word-wrap: break-word;'>Representation</td><td style='text-align: center; word-wrap: break-word;'>Clean pan</td><td style='text-align: center; word-wrap: break-word;'>Uncover &amp; pour</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>/</td><td style='text-align: center; word-wrap: break-word;'>/</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>force</td><td style='text-align: center; word-wrap: break-word;'>vector</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>4/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>impulse</td><td style='text-align: center; word-wrap: break-word;'>vector</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>7/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>force</td><td style='text-align: center; word-wrap: break-word;'>point cloud</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>6/10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>impulse</td><td style='text-align: center; word-wrap: break-word;'>point cloud</td><td style='text-align: center; word-wrap: break-word;'>7/10</td><td style='text-align: center; word-wrap: break-word;'>8/10</td></tr></table>

feedback only activates upon the contact between hands and objects, thus not aiding in the object's identification or localization. Moreover, using force data delivers slightly inferior performance compared to impulse data, possibly due to the zero-point drift in tactile readings over time. This drift, likely caused by the deformation of viscous materials used in sensor embedding, can persistently alter force reading during tasks, leading to unstable performance. Impulse, by contrast, is more robust against such variations.

### VI. CONCLUSION AND LIMITATION

Limitation. Our system has two main limitations: (i) Vision Pro's hand tracking is inaccurate when fingers are self-occluded, causing jerky control commands. This could be mitigated by fusing data from wearable devices with Vision Pro. (ii) Subtle changes in haptic feedback are difficult to sense when the robot touches an object lightly. In future work, we will consider using actuators with larger contact areas and better sensitivity, such as piezoelectric actuators, to address this problem.

Conclusion. In this paper, we introduce Bunny-VisionPro, a bimanual dexterous teleoperation system for real-time robotic control. Our system utilizes Vision Pro to track hand poses and convert them into control commands, addressing critical challenges including loop joint hand retargeting, collision avoidance, and singularity to ensure accurate, safe, and responsive operations. Moreover, we develop a haptic feedback device that delivers tactile sensations to the operator, thus enhancing control accuracy and fostering a more immersive experience.

---

## REFERENCES

[^1]: K. Hauser, “Recognition, prediction, and planning for assisted teleoperation of freeform tasks,” Autonomous Robots, vol. 35, pp. 241–254, 2013.

[^2]: N. Zhong and K. Hauser, “Attentiveness map estimation for haptic teleoperation of mobile robot obstacle avoidance and approach,” IEEE Robotics and Automation Letters, 2024.

[^3]: A. Sivakumar, K. Shaw, and D. Pathak, “Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube,” in Proceedings of Robotics: Science and Systems (RSS), 2022.

[^4]: Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.-W. Chao, and D. Fox, "Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system," in Proceedings of Robotics: Science and Systems (RSS), 2023.

[^5]: T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proceedings of Robotics: Science and Systems (RSS), 2023.

[^6]: Z. Fu, T. Z. Zhao, and C. Finn, “Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,” arXiv preprint arXiv:2401.02117, 2024.

[^7]: H. Fang, H.-S. Fang, Y. Wang, J. Ren, J. Chen, R. Zhang, W. Wang, and C. Lu, "Low-cost exoskeletons for learning whole-arm manipulation in the wild," arXiv preprint arXiv:2309.14975, 2023.

[^9]: P. Wu, Y. Shentui, Z. Yi, X. Lin, and P. Abbeel, “Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators,” arXiv preprint arXiv:2309.13037, 2023.

[^8]: A. Toedtheide, X. Chen, H. Sadeghian, A. Naceri, and S. Haddadin, “A force-sensitive exoskeleton for teleoperation: An application in elderly care robotics,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 12624–12630.

[^10]: A. Tung, J. Wong, A. Mandlekar, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese, “Learning multi-arm manipulation through collaborative teleoperation,” in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 9212–9219.

[^11]: W. Zhao, J. Chai, and Y.-Q. Xu, "Combining marker-based mcap and rgb-d camera for acquiring high-fidelity hand motion data," in Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation, 2012, pp. 33–42.

[^12]: S. Liu, H. Jiang, J. Xu, S. Liu, and X. Wang, “Semi-supervised 3d hand-object poses estimation with interactions in time,” in CVPR, 2021.

[^13]: H. Zhang, Z. Zhao, Y. Yu, K. Gui, X. Sheng, and X. Zhu, "A feasibility study on an intuitive teleoperation system combining imu with semg sensors," in Intelligent Robotics and Applications: 11th International Conference, ICIRA 2018, Newcastle, NSW, Australia, August 9–11, 2018, Proceedings, Part I 11. Springer, 2018, pp. 465–474.

[^14]: T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in ICRA, 2018.

[^15]: J. DelPreto, J. I. Lipton, L. Sanneman, A. J. Fay, C. Fourie, C. Choi, and D. Rus, "Helping robots learn: A human-robot master-apprentice model using demonstrations via virtual reality teleoperation," in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 10226–10233.

[^16]: Y. Pan, C. Chen, D. Li, Z. Zhao, and J. Hong, “Augmented reality-based robot teleoperation system using rgb-d imaging and attitude teaching device,” Robotics and Computer-Integrated Manufacturing, vol. 71, p. 102167, 2021.

[^17]: A. Iyer, Z. Peng, Y. Dai, I. Guzey, S. Haldar, S. Chintala, and L. Pinto, “Open teach: A versatile teleoperation system for robotic manipulation,” arXiv preprint arXiv:2403.07870, 2024.

[^18]: C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu, "Dexcap: Scalable and portable mocap data collection system for dexterous manipulation," arXiv preprint arXiv:2403.07788, 2024.

[^19]: H. Liu, Z. Zhang, X. Xie, Y. Zhu, Y. Liu, Y. Wang, and S.-C. Zhu, "High-fidelity grasping in virtual reality using a glove-based system," in 2019 international conference on robotics and automation (icra). IEEE, 2019, pp. 5180–5186.

[^20]: M. Mosbach, K. Moraw, and S. Behnke, “Accelerating interactive human-like manipulation learning with GPU-based simulation and high-quality demonstrations,” in Humanoids, 2022.

[^21]: M. Schwarz, C. Lenz, A. Rochow, M. Schreiber, and S. Behnke, “Nimbro avatar: Interactive immersive telepresence with force-feedback telemanipulation,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 5312–5319.

[^22]: MANUS Meta, “Manus meta quantum mocap metagloves,” <https://www.manus-meta.com/products/quantum-mocap-metagloves>.

[^23]: A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox, "Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system," in ICRA, 2020.

[^24]: S. Li, N. Hendrich, H. Liang, P. Ruppel, C. Zhang, and J. Zhang, "A dexterous hand-arm teleoperation system based on hand pose estimation and active vision," IEEE Transactions on Cybernetics, 2022.

[^25]: Y. Qin, H. Su, and X. Wang, “From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation,” RA-L, vol. 7, no. 4, pp. 10873–10881, 2022.

[^26]: S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto, “Holo-dex: Teaching dexterity with immersive mixed reality,” arXiv preprint arXiv:2210.06463, 2022.

[^27]: L. S. Yim, Q. T. Vo, C.-I. Huang, C.-R. Wang, W. McQueary, H.-C. Wang, H. Huang, and L.-F. Yu, "Wfh-vr: Teleoperating a robot arm to set a dining table across the globe via virtual reality," in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 4927–4934.

[^28]: P. Ponomareva, D. Trinitatova, A. Fedoseev, I. Kalinov, and D. Tsetserukou, “Grasplook: a vr-based telemanipulation system with r-cnn-driven augmentation of virtual environment,” in 2021 20th International Conference on Advanced Robotics (ICAR). IEEE, 2021, pp. 166–171.

[^29]: T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi, S. Levine, and J. Malik, “Learning visuotactile skills with two multifingered hands,” arXiv preprint arXiv:2404.16823, 2024.

[^30]: E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, "Bc-z: Zero-shot task generalization with robotic imitation learning," in Conference on Robot Learning. PMLR, 2022, pp. 991–1002.

[^31]: A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich, "Rt-1: Robotics transformer for real-world control at scale," arXiv preprint arXiv:2212.06817, 2022.

[^32]: A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Chormanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich, "Rt-2: Vision-language-action models transfer web knowledge to robotic control," arXiv preprint arXiv:2307.15818, 2023.

[^33]: S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Giménez, Y. Sulsky, J. Kay, J. T. Springerberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas, "A generalist agent," Transactions on Machine Learning Research, 2022, featured Certification, Outstanding Certification. [Online]. Available: <https://openreview.net/forum?id=1ikK0kHjvj>

[^34]: C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, "Diffusion policy: Visuomotor policy learning via action diffusion," in Proceedings of Robotics: Science and Systems (RSS), 2023.

[^35]: J. Pari, N. M. Shafiullah, S. P. Arunachalam, and L. Pinto, “The surprising effectiveness of representation learning for visual imitation,” in Proceedings of Robotics: Science and Systems (RSS), 2022.

[^36]: A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín, "What matters in learning from offline human demonstrations for robot manipulation," arXiv preprint arXiv:2108.03298, 2021.

[^37]: S. Haldar, J. Pari, A. Rai, and L. Pinto, “Teach a robot to fish: Versatile imitation from one minute of demonstrations,” in Proceedings of Robotics: Science and Systems (RSS), 2023.

[^38]: C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu,

---

and A. Anandkumar, “Mimicplay: Long-horizon imitation learning by watching human play,” in Proceedings of the 7th Conference on Robot Learning (CoRL), 2023.

[^39]: M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” in Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.

[^40]: Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, "3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations," in Proceedings of Robotics: Science and Systems (RSS), 2024.

[^41]: A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.-W. Chao, and D. Fox, "Rvt: Robotic view transformer for 3d object manipulation," arXiv:2306.14896, 2023.

[^42]: T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki, “Act3d: 3d feature field transformers for multi-task robotic manipulation,” in 7th Annual Conference on Robot Learning, 2023. [Online]. Available: <https://openreview.net/forum?id=-HFJuX1uqs>

[^43]: J. Grannen, Y. Wu, B. Vu, and D. Sadigh, “Stabilize to act: Learning to coordinate for bimanual manipulation,” in Conference on Robotic Learning (CoRL), 2023.

[^44]: W. Yang, A. Angleraud, R. S. Pieters, J. Pajarinen, and J.-K. Kämäräinen, “Seq2seq imitation learning for tactile feedback-based manipulation,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 5829–5836.

[^45]: Y. Park, “Teleopeation system using apple vision pro.” [Online]. Available: <https://github.com/Improbable-AI/VisionProTeleop>

[^46]: Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang, "Dexmv: Imitation learning for dexterous manipulation from human videos," in European Conference on Computer Vision. Springer, 2022, pp. 570–587.

[^47]: P. T. Boggs and J. W. Tolle, “Sequential quadratic programming,” Acta numerica, vol. 4, pp. 1–51, 1995.

[^48]: S. G. Johnson and J. Schueller, "Nlopt: Nonlinear optimization library," Astrophysics Source Code Library, pp. ascl–2111, 2021.

[^49]: A. Dietrich, T. Wimbock, A. Albu-Schaffer, and G. Hirzinger, “Reactive whole-body control: Dynamic mobile manipulation using a large number of actuated degrees of freedom,” IEEE Robotics & Automation Magazine, vol. 19, no. 2, pp. 20–33, 2012.

[^50]: M. Bhardwaj, B. Sundaralingam, A. Mousavian, N. D. Ratliff, D. Fox, F. Ramos, and B. Boots, “Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation,” in Conference on Robot Learning. PMLR, 2022, pp. 750–759.

[^51]: T. Yoshikawa, “Manipulability of robotic mechanisms,” The international journal of Robotics Research, vol. 4, no. 2, pp. 3–9, 1985.

[^52]: E. G. Gilbert, D. W. Johnson, and S. S. Keerthi, “A fast procedure for computing the distance between complex objects in three-dimensional space,” IEEE Journal on Robotics and Automation, vol. 4, no. 2, pp. 193–203, 1988.

[^53]: A. Barnea, C. Oprisan, and D. Olaru, “Force sensitive resistors calibration for use in gripping devices,” Mechanical testing and diagnosis, vol. 2, no. 3, pp. 18–27, 2012.

[^54]: G. S. Lakshmi and R. Hemavathi, “Calibration of force sensitive resistor used in force controlled grippers,” Int. J. Eng. Res. Technol, vol. 9, pp. 356–358, 2020.

[^55]: Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang, “Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation,” in Conference on Robot Learning. PMLR, 2023, pp. 594–605.