# HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments Jongbin Lim$^{1,*}$, Taeyun Ha$^{1,*}$, Seongho Cha$^{1}$, Kanghyeon Cho$^{1}$, Mingi Choi$^{1}$, Subin Jeon$^{1}$, Jisoo Kim$^{1}$, Byungjun Kim$^{1}$, and Hanbyul Joo$^{1,2}$ $^{1}$Seoul National University, $^{2}$RLWRLD Abstract—We present HRDexDB, a real-world 4D dexterous grasping dataset capturing 3D hand-object interaction trajectories over time across five embodiments. The dataset comprises 3.2K trials over 100 diverse objects. Using a synchronized multi-camera system and an integrated reconstruction pipeline, HRDexDB provides multi-view and egocentric RGB observations, 3D hand geometry, robot states, and object 6D pose trajectories, together with success/failure annotations. Human and robotic hands interact with shared objects, enabling the study of embodiment-dependent grasp strategies and contact patterns. We demonstrate the dataset's utility through human-to-robot contact map transfer, visual robot-object contact estimation, and retrieval-assisted grasping. Together, these results establish HRDexDB as a resource for studying and learning dexterous interactions across human and robotic embodiments. ### I. INTRODUCTION Enabling robots to achieve human-level dexterity is one of the central goals in robotics. Dexterous hands offer diverse ways to establish and maintain contact with objects, but their capabilities depend on morphology, kinematics, and activation. Even when grasping the same object, human and robotic hands may adopt different finger configurations and contact patterns to achieve a stable grasp. Understanding these variations is important for developing grasping methods that accommodate different hand designs and for learning from demonstrations collected across embodiments. This motivates studying how human and robotic hands interact with a common set of objects in the real world. Existing datasets provide substantial resources for studying individual aspects of dexterous manipulation, but the current data landscape remains fragmented across embodiments and annotation types. Human-centric datasets provide either isolated hand motion without objects [^1], human hand-object interactions [^2], [^3], [^4], [^5], [^6], [^7], [^8], [^9], [^10], [^11], or large-scale egocentric RGB videos without 3D information [^12], [^13]. Robotic hand datasets [^14], [^15], [^16], [^17], [^18], in contrast, focus on the robot side and often leave object motion only partially tracked. A few recent approaches attempt to connect demonstrations across embodiments [^19], [^20], [^21], yet the combination of multiple dexterous hand designs, shared object coverage, and detailed multimodal observations remains limited. These gaps make it difficult to examine embodiment-dependent grasping behavior. To address these limitations, we introduce HRDexDB, a real-world 4D dexterous grasping dataset containing 3.2K trials across five embodiments, comprising a human hand and four robotic hands, over a shared collection of 100 objects. HRDexDB captures the temporal evolution of 3D hand-object interactions through synchronized multi-view recording and reconstruction. The dataset provides reconstructed hand geometry and object 6D trajectories in a unified world coordinate system, together with success/failure annotations. HRDexDB is the first dataset to provide human and multiple robotic hands interacting with shared objects using a markerless multi-view RGB system. To relate grasping behavior across embodiments, we associate human and robot trials through a reference-based collection protocol: a teleoperator observes a human grasp and reproduces its strategy while adapting it to the target robot's morphology and kinematic constraints. These associations represent a shared grasping intent without assuming identical contacts or frame-level temporal alignment. We demonstrate the utility of HRDexDB through three downstream tasks spanning human-to-robot contact map transfer, visual contact perception, and retrieval-assisted grasp synthesis. First, we study human-to-robot contact map transfer, where a model predicts robot-specific contact patterns from a human grasp while accounting for differences in hand morphology. The transferred contacts improve grasp success over directly using human contacts in both simulation and real-world experiments. Second, we introduce visual robot-object contact estimation, using reconstructed robot geometry and tracked object poses to derive region-level contact labels for single-view RGB images, including regions occluded from the camera. Third, we study retrieval-assisted grasping, which uses local object geometry to retrieve robot grasps from our dataset and adapts them to unseen objects through geometric alignment and contact refinement. Together, these tasks demonstrate how HRDexDB's demonstration-execution associations, geometric annotations, and recorded robot grasps support downstream robotics applications. In summary, our contributions are threefold: (1) We introduce HRDexDB, a real-world dataset of human and robotic dexterous grasping over shared objects, with multimodal observations and detailed 3D annotations. (2) We present a unified pipeline for synchronized multi-view capture and 3D reconstruction of human and robot hand-object interactions. (3) We establish downstream benchmarks on HRDexDB, --- ![Image](images/c5c3f2a2091882943f49a8ccdd293716ff08757e0989b563719e4cf655b5de57.jpg)
Fig. 1: Overview of HRDexDB. HRDexDB contains 3.2K human and robotic dexterous grasping episodes across 100 objects and multiple hand embodiments. Using a synchronized multi-view capture system, we record human demonstrations and robotic executions with reconstructed 3D hand and robot trajectories, object 6D poses, egocentric observations, and success/failure annotations.
including human-to-robot contact map transfer, RGB to robot-object binary contact estimation, and retrieval-assisted grasping. At the time of submission, HRDexDB includes 100 captured objects, with ongoing expansion toward 1,000 objects. The full dataset and benchmark protocols will be publicly released to facilitate future research in dexterous manipulation. ### II. RELATED WORK Human-Object Interaction Dataset. Human-object interaction datasets have enabled substantial progress in modeling hand articulation and object motion under contact-rich manipulation. Benchmarks such as FreiHAND [^4], HO-3D [^11], and DexYCB [^6] focused on 3D hand-object pose estimation, while other datasets such as ContactPose [^5], ARCTIC [^8], HOT3D [^25], HOI4D [^7], GigaHands [^26], TACO [^23], Contact4D [^24], OakInk2 [^9], and HO-Cap [^10] expanded the scale, viewpoints, object diversity, contact annotations, and task complexity of hand-object interaction capture. These datasets provide rich observations and annotations for understanding human hand-object interactions. Complementing this line of work, HRDexDB captures both human and robotic grasping over shared objects within a common calibrated setup, supporting the study of how grasp configurations and contact patterns vary across embodiments. Robot-Object Interaction Dataset. Robot manipulation datasets have grown substantially in scale and diversity for robot learning. Large-scale efforts such as Open X-Embodiment [^15] and DROID [^17] aggregate diverse demonstrations across many tasks and environments, but much of this data is collected with relatively low-DoF grippers. More recent datasets such as AgiBot World [^14], RoboMIND [^18], and RoboCOIN [^16] further expand scale and task diversity, supporting bimanual manipulation and demonstrations collected with both grippers and dexterous hands. Focusing on dexterous grasping, RealDex [^27] collects real-world dexterous robot grasping trajectories through teleoperation, but focuses on a single robotic hand. HRDexDB complements these resources by providing detailed 3D annotations of both robotic hand motion and object pose trajectories across multiple dexterous hand embodiments, together with calibrated multi-view RGB observations. These annotations support geometric analysis of robot-object interactions throughout the grasping sequence. Human-Robot Demonstration Dataset. Recent datasets aim to associate human demonstrations with robotic executions to support learning across embodiments. RH20T [^19] and H&R [^21] provide task- or frame-level human-robot ---
TABLE I: Comparison of Human-Object Interaction (HOI) and Robotics Datasets with HRDexDB
DatasetType#Emb.DexRobotHandModalityViewsObjsResolutionSeqsFramesMarkerless3DHandObj6D
FPHA[^2]HOI1--RGB-D1261920 \times 10801.2K105K\times✓✓✓✓✓
ContactDB[^3]HOI1--RGB-D9501920 \times 10803.7K375K✓✗✗✓✓✓
FreiHAND[^4]HOI1--RGB8251280 \times 1024-37K✓✓✓✓✓✓
HO-3D[^11]HOI1--RGB-D510640 \times 48068103K✓✓✓✓✓✓
GRAB[^22]HOI1--MoCap-51-1.3K1.6M\times✓✓✓✓✓
ContactPose[^5]HOI1--RGB-D325960 \times 5402.3K2.9M\times✓✓✓✓✓
DexYCB[^6]HOI1--RGB-D820640 \times 4801K582K✓✓✓✓✓✓
HOI4D[^7]HOI1--RGB-D18001280 \times 7204K2.4M✓✓✓✓✓✓
ARCTIC[^8]HOI1--RGB9112800 \times 20003392.1M\times✓✓✓✓✓
TACO[^23]HOI1--RGB/RGB-D131964096 \times 30002.5K5.2M\times✓✓✓✓✓
OakInk2[^9]HOI1--RGB-D1675840 \times 4806271.34M\times✓✓✓✓✓
Contact4D[^24]HOI1--RGB19N.A.3840 \times 21603752M\times✓✓✓✓✓
HOT3D[^25]HOI1--RGB2331408 \times 14084253.7M\times✓✓✓✓✓
GigaHands[^26]HOI1--RGB514171280 \times 72014K183M✓✓✓✓✓✓
HO-Cap[^10]HOI1--RGB-D964640 \times 48064656K✓✓✓✓✓✓
RealDex[^27]ROI1✓✓RGB-D452-2.6K955K✓✓✓✓✓✓
RoboCOIN[^16]ROI15✓✓RGB-D3432-180K-✓✓✓✓✓✓
AgiBotWorld[^14]ROI1✓✓RGB-D83000-1M-✓✓✓✓✓✓
OXE[^15]ROI22\times-RGB-D3--1M130M✓✓✓✓✓✓
DROID[^17]ROI1\times-RGB-D3-1280 \times 72076K56.7M✓✓✓✓✓✓
RoboMIND[^18]ROI4✓-RGB-D396480 \times 640107K-✓✓✓✓✓✓
RH20T[^19]HROI7\times-RGB-D7-1280 \times 720220K50M✓✓✓✓✓✓
DexWild[^20]HROI2✓-RGB-D6180224 \times 22410K-✓✓✓✓✓✓
H&R[^21]HROI2\times-RGB-D1-240 \times 4242.6K1M✓✓✓✓✓✓
HRDexDB (Ours)HROI5✓✓RGB231002048 \times 15363.2K43.5M✓✓✓✓✓✓
alignment, but are primarily based on parallel-jaw grippers and offer limited coverage of multi-finger dexterous manipulation. DexWild [^20] scales in-the-wild dexterous human demonstrations and aligns human and robot observation and action spaces for co-training, without establishing reference-linked correspondences between individual executions. In HRDexDB, human demonstrations guide the collection of robot grasps, while allowing hand-specific configurations and contact patterns. The resulting demonstration-execution associations support downstream studies such as contact map transfer, without assuming identical contacts or frame-level temporal alignment across embodiments. Synthetic Dexterous Grasp Datasets. Large-scale synthetic grasp datasets provide a complementary direction for learning dexterous manipulation priors. DexGraspNet [^28] generates stable dexterous grasps in simulation, while GenDex-Grasp/MultiDex [^29] and CEDex [^30] extend grasp synthesis to multiple robotic hand embodiments. GraspXL [^31] scales grasp motion generation across human and robotic hands. While synthetic datasets offer substantially greater scale, their generated grasps are not necessarily validated through physical execution. HRDexDB provides recorded real-world trials in which successful grasps are verified by lifting the object with the physical robot, offering empirical evidence of grasp feasibility under the recorded execution conditions. ### III. CONSTRUCTING HRDEXDB ### A. Dataset Overview HRDexDB is constructed as a multi-modal dataset of dexterous grasping sequences performed by both a human subject and four robotic embodiments, spanning Allegro Hand V5 Plus, Inspire Hand RH56DFTP and RH56F1, and Robotiq 2F-85 gripper. The dataset contains 43.5M frames and 3.2K sequences over 100 objects, including synchronized visual observations, kinematic states, reconstructed geom- ![Image](images/c128533ac2858dd8056aa20d62ea33beb4fdb2ff3ad20f8ed4536de36695e9ca.jpg)
Fig. 2: Demonstration-guided data acquisition. A human grasp serves as a reference for teleoperated robot executions, with grasp configurations adapted to each embodiment.
try, object 6D poses, and tactile signals when available. All spatial quantities are aligned in a unified world coordinate system defined by our calibrated multi-camera platform. A human grasping trial is represented as a time-indexed sequence $$ \begin{array}{r}{\mathcal{T}^{\mathrm{h u m a n}}=\left\{\{\mathbf{I}_{t}^{c_{i}}\}_{i=1}^{21},\mathbf{I}_{t}^{\mathrm{e g o}},\mathbf{\theta}_{t}^{\mathrm{h u m a n}},T_{t}^{\mathrm{o b j e c t}},y\right\}_{t=1}^{T_{h}}.}\end{array} $$ Here, $\{\mathbf{I}_t^{ci}\}_{i=1}^{21}$ and $\mathbf{I}_t^{\text{ego}}$ denote synchronized exocentric and egocentric RGB observations, respectively. $\theta_t^{\text{human}} \in \mathbb{R}^{51}$ denotes the MANO parameters, $\mathbf{T}_t^{\text{object}} \in \text{SE}(3)$ is the object 6D pose, and $y \in \{0,1\}$ indicates grasp success. $T_h$ denotes the human sequence length. Similarly, a robotic grasping trial is represented as $$ \mathcal{T}^{\mathrm{r o b o t}}=\left\{\{\mathbf{I}_{t}^{c_{i}}\}_{i=1}^{21},\mathbf{I}_{t}^{\mathrm{e g o}},\boldsymbol{q}_{t}^{\mathrm{r o b o t}},\boldsymbol{T}_{t}^{\mathrm{o b j e c t}},y\right\}_{t=1}^{T_{r}}, $$ where $q_{t}^{robot}$ denotes the recorded robot states and commands and $T_{r}$ is the robot sequence length. Each robot trial additionally includes metadata identifying the human demonstration used as its reference during collection. This association does not imply frame-level temporal alignment between the two executions. Tactile signals $F_{t}^{tactile}$ are provided as an additional modality for robots equipped with fingertip tactile sensors. --- ![Image](images/7f1053024ee96c700fbee11e8ec1ac73c5abf3b6155e1738e3f2e51ad4c0dfeb.jpg)
Fig. 3: Capture and Reconstruction Pipeline. Multi-view recordings are used to reconstruct hand motion and 6D object trajectories, yielding spatially aligned and temporally synchronized 3D annotations.
### B. Data Acquisition Protocol We collect human and robot grasping trials through a two-stage procedure using the same target object and workspace setup. A human participant first demonstrates a grasp, which is recorded by the multi-view capture system. A teleoperator then observes the demonstration and attempts to reproduce its grasping strategy with the target robotic hand. Notably, not every human grasp can be reproduced by every robotic hand due to differences in morphology and kinematic constraints. Human demonstrations therefore serve as references, with robot executions allowing embodiment-specific adaptations in finger configuration and contact placement. Each robot trial is recorded as a separate execution with its own motion trajectory and success/failure outcome. We retain the association between each robot trial and its reference human demonstration as collection metadata. ### C. Multi-Modal State Reconstruction Capture System. Our capture platform (Fig. 3) combines a 21-camera RGB rig surrounding the workspace with a stereo helmet for human demonstrations and static stereo egocentric cameras for robotic trials. Camera intrinsics and extrinsics are calibrated using ChArUco observations and COLMAP [^32], with hand-eye calibration aligning the rig and robot frames. For human sequences, moving egocentric camera poses are recovered through ego-exo feature matching and PnP. Each robotic hand is mounted on an xArm6 manipulator and teleoperated using a Vive wrist tracker and MANUS gloves for wrist and finger control, respectively. Human Hand Reconstruction. To reconstruct 3D human hand motion, we employ the MANO parametric hand model [^33]. Following the multi-view fitting strategy of GigaHands [^26], we detect 2D hand keypoints in each calibrated view using HaMeR [^34], triangulate 3D joints, and optimize MANO pose parameters for each frame. Subject-
TABLE II: Trial counts and success rates by embodiment.
EmbodimentTrialsSuccessesFailuresSuccess (%)
Human5415410100.00
Allegro V55394875290.35
Inspire RH56F174351323069.04
Inspire RH56DFTP79147631560.18
Robotiq 2F-8560147412778.87
Robot subtotal2,6741,95072472.92
Total3,2152,49172477.48
specific hand shape is calibrated using silhouette alignment with SAM3-generated masks [^35], and temporal filtering is applied to reduce jitter. Robot Hand Reconstruction. We reconstruct the robot hand geometry through forward kinematics using the recorded joint configurations and the corresponding URDF model. The resulting articulated meshes are transformed into the calibrated world frame using the recorded robot poses. Object 6D Tracking. We estimate object 6D pose trajectories using a model-based multi-view tracking pipeline. Given multi-view RGB observations, SAM3 [^35] provides object masks at a selected initialization frame to localize the manipulated object. We then estimate its initial 6D pose using FoundPose [^36] with the object's 3D CAD model, and track the object across the full sequence using GoTrack [^37]. At each frame, observations from the selected calibrated views are jointly fused through multi-view triangulation and rigid pose fitting, improving robustness to severe hand-object occlusions and viewpoint-specific tracking failures. ### IV. DATASET ANALYSIS ### A. Dataset Statistics HRDexDB comprises 3,215 grasping trials, including 541 human demonstrations and 2,674 robot executions across --- ![Image](images/5c205f2a726d07dee447e8ec2198428cf6d802b46d866653b6af0f9c12b319d8.jpg)
(b) Object diversity
Fig. 4: Grasp and object diversity in HRDexDB. (a) Human and robotic grasps, colored by grasp type. (b) Object distribution by OBB volume, aspect ratio, and mass. Aspect ratio is the ratio of the longest to shortest OBB dimension.
TABLE III: Object silhouette consistency (mean IoU $\uparrow$) against manually verified SAM3 masks.
EmbodimentFoundationPoseOurs
Human0.8410.915
Allegro V50.7420.905
Inspire RH56F10.8940.942
Inspire RH56DFTP0.7230.908
Robotiq 2F-850.8050.933
Overall0.8010.921
four embodiments. Table II summarizes the trial counts and outcomes for each embodiment. All human demonstrations are successful, while robot trials include both successful and failed executions. We observe that the limited thumb mobility of the Inspire hands makes certain reference pinch grasps difficult to reproduce during teleoperation, even for experienced teleoperators, which may partly explain their lower success rates compared with the other embodiments. ### B. Grasp and Object Diversity HRDexDB covers 23 of the 33 grasp types defined by the GRASP taxonomy [^38], including Precision Sphere and Palmar Pinch. Figure 4(a) illustrates this diversity through representative human and robotic grasps. We will release these grasp-type labels with the dataset. The dataset includes 100 everyday objects spanning kitchenware, household items, toys, tools, and organic objects. Figure 4(b) shows their distribution in OBB volume, aspect ratio, and mass. OBB volumes range from 0.11 to 8.70 L (median 1.31 L), while masses range from 3 to 533 g (median 107 g). ### C. Object 6D Pose Tracking Quality We evaluate our 6D object pose tracking by measuring image-space consistency on 1,500 images, 300 per embodiment (Table III). We compare our multi-view GoTrack pipeline with a two-view FoundationPose [^39] tracking baseline, using identical evaluation images and occlusion handling. We compute IoU between depth-aware rendered object silhouettes and manually verified SAM3 masks in the capture views used for annotation. Our pipeline achieves a mean IoU of 0.921, compared with 0.801 for FoundationPose, with improvements across all five embodiments. ### V. EXPERIMENTS & BENCHMARKING We evaluate HRDexDB through three downstream tasks: human-to-robot contact map transfer, visual robot-object contact estimation, and retrieval-assisted grasping. ### A. Human-to-Robot Contact Map Transfer Task definition. Human contact patterns provide useful references for robot grasping, but differences in hand morphology and kinematics can make direct imitation suboptimal. We investigate whether HRDexDB enables learning robot-specific contact representations from human grasps. Given an object and an observed human contact representation, the task is to predict contact and hand-part maps for a target robotic hand, which subsequently guide grasp synthesis. Experimental setup. We use the demonstration-execution associations in HRDexDB to supervise human-to-robot contact map transfer. The associated robot contact map serves as the target for each reference human grasp, allowing embodiment-specific differences in contact placement and finger configuration. We represent a grasp on an object point cloud $O \in \mathbb{R}^{N \times 3}$ with a contact map $C \in [0, 1]^N$ of per-point contact probabilities and a part map P assigning contacted points to hand parts. The human part map is $P^h \in \mathbb{R}^{N \times 6}$, while the robot part map is $P^r \in \mathbb{R}^{N \times B}$, with $B = 6$ for the Inspire hand and $B = 5$ for the Allegro hand. Conditioned on the human representation $[C^h, P^h]$ and PointNet++ [^40] object features, our model predicts the robot representation $[C^r, P^r]$, supervised with a contact-weighted $L_1$ loss on $C^r$ and a cross-entropy loss on $P^r$ over contacted points. We train a separate model for each target hand, using the Inspire RH56F1 and Allegro Hand V5 Plus. Given the predicted robot contact representation, we synthesize grasps using the physics-aware optimization pipeline of CEDex [^30], which combines contact, penetration, and self-collision terms. We compare Human-Contact, which optimizes against the captured human contact map, with Transferred-Contact, which optimizes against our predicted robot-specific contact map. The optimizer is fixed across both settings, isolating the effect of the contact objective. For the four-fingered Allegro hand, the baseline maps the fourth finger to the union of the human ring and little fingers. We evaluate real-world grasp success using held-out human grasps of objects seen during training as inputs. A grasp --- ![Image](images/20cccadf041e9a44a401c9c1ec4abda2c1ceff7f0362616b5e48336db876744b.jpg)
(a) Human-to-robot contact transfer
(b) Real-world execution
Fig. 5: Human-to-robot contact transfer and real-world grasping. (a) Human contact and part maps ( $C^{h}$, $P^{h}$) are transferred to robot-specific maps ( $\hat{C}^{r}$, $\hat{P}^{r}$) for grasp optimization. (b) For the same object and human grasp, transferred contacts yield a successful grasp, whereas direct human contacts fail.
TABLE IV: Real-world grasp success rates (%) for human-to-robot contact map transfer, evaluated over 60 trials per hand and condition. Human and transferred contact maps guide the same grasp optimizer.
MethodInspire $\uparrow$Allegro $\uparrow$
Human-Contact66.760.0
Transferred (Ours)73.370.0
Improvement+6.7+10.0
TABLE V: Contact transfer generalization. Grasp success rate (%) on unseen objects.
MethodDexYCB (20 obj.s.)GRAB (48 obj.s.)
Human-Contact53.6950.74
Transferred (Ours)55.9456.81
Improvement+2.25+6.07
succeeds if the object is lifted and held for ten seconds. We generate pre-grasp and squeeze motions with BODex [^41] and compute execution trajectories with CuRobo [^42]. Results. Figure 5 illustrates how transferred contact maps adapt the human contact distribution to the target robot hand. Table IV summarizes grasp success rates under the two contact objectives. Transferred-Contact improves success over directly using human contact maps, showing that HRDexDB enables learning robot-specific contact strategies from demonstration guided robot grasps. We further evaluate contact transfer with Allegro V5 on 20 unseen objects from DexYCB [^6] and 48 from GRAB [^22] in simulation. For each object, we use 10 distinct human grasps as inputs and generate a robot grasp with each of eight random seeds per input, yielding 80 robot grasps per object. Following CEDex, we apply forces along the positive and negative directions of the three coordinate axes in Isaac Gym [^43]. Transferred contact outperforms human contact on both datasets (Table V), demonstrating that contact transfer learned from HRDexDB generalizes to unseen objects. ### B. Visual Robot-Object Contact Estimation Task definition. Tactile sensing is an essential component of dexterous manipulation [^44], [^45], [^46], yet tactile cues are largely absent from existing robot manipulation methods because reliable, readily available sensors are scarce. Recently, vision-based contact and pressure estimation has been explored for human hands [^47]. We adopt an analogous ![Image](images/57e779735869a7b1cb22679ef6acd6df42c0f9239b6e8dff33b1a99e5a45dd0b.jpg)
Fig. 6: Geometry-derived contact annotation. From left to right: RGB observations, reconstructed hand-object geometry, and region-level contact labels for Inspire and Allegro hands. Regions within 3 mm of the object surface are labeled as contacting and highlighted in red.
approach for robot hands and predict binary contact directly from visual observations, which can potentially reduce the need to design, mount, and calibrate tactile sensors for each robot hand. We benchmark binary contact prediction for 60 hand regions from a single RGB frame on the Inspire RH56F1 and Allegro V5 hands, using hand-specific partitions that are finer near the fingertips and distal links. Contact annotation. $^{1}$ As illustrated in Figure 6, we derive region-level contact labels from the reconstructed 4D hand-object geometry. A hand region is labeled as contacting when its distance to the object surface is at most 3 mm. This provides supervision for both visible and occluded hand regions in the input RGB image. Baselines and experimental setup. We adapt HOPE [^47], HACO [^48], and PressureVision [^49] to predict 60 region-level contact labels. For HOPE, we replace the human-hand vertex tokens with 60 robot surface-region tokens conditioned on their articulated palm frame positions and the camera-relative palm pose, removing temporal attention and pressure prediction for single-frame binary contact estimation. For HACO, we replace the MANO-based [^33] vertex outputs and multi-resolution contact mappings with 60 robot surface-region outputs. PressureVision uses robot state to project the hand mesh for dense supervision and region-level prediction. At inference, we average logits sampled at each region's projected vertices, treating samples outside the image or behind the camera as non-contact. Inspire RH56F1 includes 74/18 objects and 56.1K/14.6K images for training/validation; Allegro V5 includes 72/19 objects and 37.3K/8.2K images. HOPE, HACO, and PressureVision are trained for 20, 10, and 60 epochs, respectively. We report micro-averaged precision, recall, and F1 over all valid validation frame-region pairs. $^{1}$The Inspire and Allegro hands are equipped with tactile sensors, but their readings are too noisy or sparse to serve as reliable contact labels. We still release these signals with HRDexDB. ---
TABLE VI: Region-level contact estimation on two robot-hand embodiments (Inspire RH56F1/Allegro V5).
EmbodimentMethodInputF1PrecisionRecall
Inspire RH56F1PressureVisionRGB + robot state0.5940.7910.476
HACORGB0.4140.2780.810
HOPERGB + robot state0.6790.6890.669
Allegro V5PressureVisionRGB + robot state0.4240.7070.303
HACORGB0.4760.3380.806
HOPERGB + robot state0.6440.6810.612
![Image](images/9782bbb0e4e4625a4bd7c29eca9bb6061126708c4887a6f9a1a7e79bb4832835.jpg)
Fig. 7: Retrieval-Assisted Grasping. Given a human grasp, we retrieve robot grasps from HRDexDB using local object geometry, align them to the target object, and refine hand-object contacts to generate a robot grasp.
Results. HACO achieves high recall but low precision, whereas PressureVision shows the opposite trade-off. HOPE achieves the highest F1 score on both embodiments, with a balanced precision-recall trade-off. These comparisons reflect the adapted pipelines, including differences in auxiliary inputs and region-level prediction procedures. Overall, the results establish baselines for contact estimation on unseen objects and demonstrate how HRDexDB's geometric annotations enable supervision across robotic-hand embodiments. ### C. Retrieval-Assisted Grasping Task definition. Human grasps provide a natural prior for where and how a robot should grasp an object. However, kinematic retargeting of human grasps may fail to establish stable contacts due to differences in hand morphology and kinematics. We investigate whether robot grasps recorded in HRDexDB can complement retargeted grasps with morphology-compatible priors. Given a human grasp and the target object mesh, we formulate Retrieval-Assisted Grasping (RAG), which generates a robot grasp near the demonstrated one by retrieving and adapting database grasps with similar local object geometry. Method. RAG combines geometric retrieval, local alignment, and contact refinement (Fig. 7). We first obtain a robot wrist pose $\mathbf{T}_q$ from the human grasp using AnyTeleop [^50]. Inspired by hand-centric object representation of Robust-DexGrasp [^51], we construct a local geometry descriptor for grasp retrieval. We place the robot hand at $\mathbf{T}_q$ with a fixed canonical finger configuration $\mathbf{q}_{\text{can}}$, sample $N_h = 168$ hand surface points, and sample $N_o = 4096$ area-weighted points from the object mesh. For each hand point $\mathbf{x}_i$, we find the nearest sampled object point $\mathbf{y}_i$ and construct a descriptor from the relative vectors $\mathbf{y}_i - \mathbf{x}_i$ and their Euclidean distances. We retrieve the top 16 database grasps and process them in ranked order. For each candidate, we extract object patches within 4 cm of the hand surface and align the database patch
TABLE VII: Grasping success rate on 20 unseen objects.
MethodSuccess Rate $\uparrow$
Sim.Real
Retargeting + Opt.0.500.40
RAG (Ours)0.730.75
Improvement+0.23+0.35
to the query patch using FPFH [^52] matching, RANSAC, and point-to-plane ICP. We transform the database wrist pose using the estimated alignment while retaining its finger configuration. We retain up to six candidates whose aligned wrist poses lie within 2 cm and $15^{\circ}$ of $T_{q}$. For each retained candidate, we identify contact anchors as hand surface points within 5 mm of the database object. We jointly refine the wrist pose and finger configuration to bring these anchors into contact with the query object, while penalizing object penetration and hand self-collision. The optimized candidate with the most hand-object contacts is selected for execution. Experimental setup. To isolate the contribution of retrieval, we compare our RAG against Kinematic Retargeting + Optimization, which applies the same contact-aware optimization used in RAG directly to the kinematically retargeted grasp, without database retrieval. We use the Allegro V5 hand for all experiments. For evaluation, we split the 100 objects in HRDexDB into 80 database objects and 20 unseen test objects. The database objects are used only for grasp retrieval, while evaluation is restricted to the 20 unseen objects. In simulation, we evaluate five human grasps per test object in MuJoCo [^53], resulting in 100 trials in total. For real-world evaluation, we use one human grasp per object, resulting in 20 trials in total. In both settings, a trial is considered successful if the robot successfully grasps and lifts the object. Results. Table VII shows that RAG improves simulated grasp success from 50/100 (50%) to 73/100 (73%) and real-world success from 8/20 (40%) to 15/20 (75%). These results support the utility of robot grasp priors retrieved from HRDexDB for grasp synthesis on unseen objects, beyond refining kinematically retargeted grasps alone, and highlighting the potential of retrieval-based priors for improving grasp generalization to novel object geometries. ### VI. CONCLUSION We present HRDexDB, a real-world 4D dataset of human and robotic dexterous grasping across multiple embodiments, comprising 3.2K sequences over 100 objects with detailed 3D hand and 6D object annotations. By capturing diverse hands interacting with shared objects, HRDexDB supports the study of embodiment-dependent grasp strategies and contact patterns. Our experiments demonstrate its utility for cross-embodiment contact map transfer, visual contact estimation, and retrieval-assisted grasping. The dataset and benchmark protocols will be publicly released. Moving forward, we plan to expand HRDexDB toward 1,000 objects and more complex functional manipulation tasks. --- ## REFERENCES [^1]: G. Moon, S.-I. Yu, H. Wen, T. Shiratori, and K. M. Lee, “Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image,” in ECCV, 2020. [^2]: G. Garcia-Hernando, S. Yuan, S. Baek, and T.-K. Kim, "First-person hand action benchmark with rgb-d videos and 3d hand pose annotations," in CVPR, 2018. [^3]: S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays, “Contactdb: Analyzing and predicting grasp contact via thermal imaging,” in CVPR, 2019. [^4]: C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox, "Freihand: A dataset for markerless capture of hand pose and shape from single rgb images," in ICCV, 2019. [^5]: S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays, "ContactPose: A dataset of grasps with object contact and hand pose," in ECCV, 2020. [^6]: Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield et al., “Dexycb: A benchmark for capturing hand grasping of objects,” in CVPR, 2021. [^7]: Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi, “Hoi4d: A 4d egocentric dataset for category-level human-object interaction,” in CVPR, 2022. [^8]: Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges, “Arctic: A dataset for dexterous bimanual hand-object manipulation,” in CVPR, 2023. [^9]: X. Zhan, L. Yang, Y. Zhao, K. Mao, H. Xu, Z. Lin, K. Li, and C. Lu, "Oakink2: A dataset of bimanual hands-object manipulation in complex task completion," in CVPR, 2024. [^10]: J. Wang, Q. Zhang, Y.-W. Chao, B. Wen, X. Guo, and Y. Xiang, "HO-cap: A capture system and dataset for 3d reconstruction and pose tracking of hand-object interaction," in NeurIPS, 2025. [^11]: S. Hampali, M. Rad, M. Oberweger, and V. Lepetit, “Honnotate: A method for 3d annotation of hand and object poses,” in CVPR, 2020. [^12]: K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al., “Ego4d: Around the world in 3,000 hours of egocentric video,” in CVPR, 2022. [^13]: D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, "The epic-kitchens dataset: Collection, challenges and baselines," IEEE TPAMI, vol. 43, no. 11, pp. 4125–4141, 2021. [^14]: Q. Bu, J. Cai, L. Chen, X. Cui, Y. Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang et al., “Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems,” arXiv preprint arXiv:2503.06669, 2025. [^15]: A. O'Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain et al., "Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0," in ICRA, 2024. [^16]: S. Wu, X. Liu, S. Xie, P. Wang, X. Li, B. Yang, Z. Li, K. Zhu, H. Wu, Y. Liu et al., “Robocoin: An open-sourced bimanual robotic data collection for integrated manipulation,” arXiv preprint arXiv:2511.17441, 2025. [^17]: A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Sriram, L. Y. Chen, K. Ellis et al., "Droid: A large-scale in-the-wild robot manipulation dataset," in RSS, 2024. [^18]: K. Wu, C. Hou, J. Liu, Z. Che, X. Ju, Z. Yang, M. Li, Y. Zhao, Z. Xu, G. Yang et al., “Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation,” in RSS, 2025. [^19]: H.-S. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu, "Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot," in ICRA, 2024. [^20]: T. Tao, M. K. Srirama, J. J. Liu, K. Shaw, and D. Pathak, “Dexwild: Dexterous human interactions for in-the-wild robot policies,” in RSS, 2025. [^21]: S. Xie, H. Cao, Z. Weng, Z. Xing, H. Chen, S. Shen, J. Leng, Z. Wu, and Y.-G. Jiang, “Human2robot: Learning robot actions from paired human-robot videos,” in AAAI, 2026. [^22]: O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas, “Grab: A dataset of whole-body human grasping of objects,” in ECCV, 2020. [^23]: Y. Liu, H. Yang, X. Si, L. Liu, Z. Li, Y. Zhang, Y. Liu, and L. Yi, "Taco: Benchmarking generalizable bimanual tool-action-object understanding," in CVPR, 2024. [^24]: J.-T. Song, J. Kim, J. Cao, Y. Lei, T. Yagi, and K. Kitani, “Contact4d: A video dataset for whole-body human motion and finger contact in dexterous operations,” in 3DV, 2026. [^25]: P. Banerjee, S. Shkodrani, P. Moulon, S. Hampali, S. Han, F. Zhang, L. Zhang, J. Fountain, E. Miller, S. Basol et al., "Hot3d: Hand and object tracking in 3d from egocentric multi-view videos," in CVPR, 2025. [^26]: R. Fu, D. Zhang, A. Jiang, W. Fu, A. Funk, D. Ritchie, and S. Sridhar, "Gigahands: A massive annotated dataset of bimanual hand activities," in CVPR, 2025. [^27]: Y. Liu, Y. Yang, Y. Wang, X. Wu, J. Wang, Y. Yao, S. Schwertfeger, S. Yang, W. Wang, J. Yu, X. He, and Y. Ma, “RealDex: Towards human-like grasping for robotic dexterous hand,” in IJCAI, 2024. [^28]: R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang, "Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation," in ICRA, 2023. [^29]: P. Li, T. Liu, Y. Li, Y. Geng, Y. Zhu, Y. Yang, and S. Huang, “Gendexgrasp: Generalizable dexterous grasping,” in ICRA, 2023. [^30]: Z. Wu, R. A. Potamias, X. Zhang, Z. Zhang, J. Deng, and S. Luo, "Cedex: Cross-embodiment dexterous grasp generation at scale from human-like contact representations," in ICRA, 2026. [^31]: H. Zhang, S. Christen, Z. Fan, O. Hilliges, and J. Song, “GraspXL: Generating grasping motions for diverse objects at scale,” in ECCV, 2024. [^32]: J. L. Schönberger and J.-M. Frahm, "Structure-from-motion revisited," in CVPR, 2016. [^33]: J. Romero, D. Tzionas, and M. J. Black, “Embodied hands: Modeling and capturing hands and bodies together,” ACM TOG (SIGGRAPH Asia), vol. 36, no. 6, pp. 245:1–245:17, Nov. 2017. [^34]: G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik, "Reconstructing hands in 3d with transformers," in CVPR, 2024. [^35]: N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang et al., “Sam 3: Segment anything with concepts,” arXiv preprint arXiv:2511.16719, 2025. [^36]: E. P. Örnek, Y. Labbé, B. Tekin, L. Ma, C. Keskin, C. Forster, and T. Hodağ, “Foundpose: Unseen object pose estimation with foundation features,” in ECCV, 2024. [^37]: V. N. Nguyen, C. Forster, B. Tekin, S. Shkodrani, V. Lepetit, C. Keskin, and T. Hodağ, "Gotrack: Generic 6dof object pose refinement and tracking," in CVPRW, 2025. [^38]: T. Feix, J. Romero, H.-B. Schmiedmayer, A. M. Dollar, and D. Kragic, "The grasp taxonomy of human grasp types," IEEE Transactions on human-machine systems, vol. 46, no. 1, pp. 66–77, 2016. [^39]: B. Wen, W. Yang, J. Kautz, and S. Birchfield, "Foundationpose: Unified 6d pose estimation and tracking of novel objects," in CVPR, 2024. [^40]: C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in NeurIPS, 2017. [^41]: J. Chen, Y. Ke, and H. Wang, “Bodex: Scalable and efficient robotic dexterous grasp synthesis using bilevel optimization,” in ICRA, 2025. [^42]: B. Sundaralingam, A. Murali, and S. Birchfield, “curobov2: Dynamics-aware motion generation with depth-fused distance fields for high-dof robots,” arxiv preprint arXiv:2603.05493, 2026. [^43]: V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance gpu-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470, 2021. [^44]: K.-W. Lee, Y. Qin, X. Wang, and S.-C. Lim, “Dextouch: Learning to seek and manipulate objects with tactile dexterity,” IEEE RA-L, vol. 9, no. 12, pp. 10772–10779, 2024. [^45]: L. Mack, F. Grüninger, B. A. Richardson, R. Lendway, K. J. Kuchenbecker, and J. Stueckler, “Visuo-tactile object pose estimation for a multi-finger robot hand with low-resolution in-hand tactile sensing,” in ICRA, 2025. [^46]: E. Miller, T. McInroe, D. Abel, O. Mac Aodha, and S. Vijayakumar, “Enhancing tactile-based reinforcement learning for robotic control,” in NeurIPS, 2025. [^47]: S. Jeon, B. Kim, and H. Joo, “Hope: Hand-object pressure estimation from monocular videos,” in arXiv preprint arXiv:2608.06192, 2026. [^48]: D. S. Jung and K. M. Lee, “Learning dense hand contact estimation from imbalanced data,” in NeurIPS, 2025. [^49]: P. Grady, C. Tang, S. Brahmbhatt, C. D. Twigg, C. Wan, J. Hays, and C. C. Kemp, “Pressurevision: estimating hand pressure from a single rgb image,” in ECCV, 2022. --- [^50]: Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.-W. Chao, and D. Fox, “Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system,” in RSS, 2023. [^51]: H. Zhang, Z. Wu, L. Huang, S. Christen, and J. Song, “RobustDex-Grasp: Robust dexterous grasping of general objects,” in Conference on Robot Learning (CoRL), 2025. [^52]: R. B. Rusu, N. Blodow, and M. Beetz, "Fast point feature histograms (fpfh) for 3d registration," in ICRA, 2009. [^53]: E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in IROS, 2012.