# HD-EPIC: A Highly-Detailed Egocentric Video Dataset Toby Perrett $^{**}$ Ahmad Darkhalil $^{**}$ Saptarshi Sinha $^{**}$ Omar Emara $^{**}$ Sam Pollard $^{**}$ Kranti Kumar Parida $^{**}$ Kaiting Liu, $^{**}$ Prajwal Gatti $^{**}$ Siddhant Bansal $^{**}$ Kevin Flanagan $^{**}$ Jacob Chalk $^{**}$ Zhifan Zhu $^{**}$ Rhodri Guerrier $^{**}$ Fahd Abdelazim $^{**}$ Bin Zhu $^{**}$ Davide Moltisanti Michael Wray Hazel Doughty Dima Damen $^{*}$Uni. of Bristol $^{*}$Leiden Uni. Singapore Management Uni. Uni. of Bath $^{*}$: Equal Contribution ## Abstract We present a validation dataset of newly-collected kitchen-based egocentric videos, manually annotated with highly detailed and interconnected ground-truth labels covering: recipe steps, fine-grained actions, ingredients with nutritional values, moving objects, and audio annotations. Importantly, all annotations are grounded in 3D through digital twinning of the scene, fixtures, object locations, and primed with gaze. Footage is collected from unscripted recordings in diverse home environments, making HD EPIC the first dataset collected in-the-wild but with detailed annotations matching those in controlled lab environments. We show the potential of our highly-detailed annotations through a challenging VQA benchmark of 26K questions assessing the capability to recognise recipes, ingredients, nutrition, fine-grained actions, 3D perception, object motion, and gaze direction. The powerful long-context Gemini Pro only achieves 37.6% on this benchmark, showcasing its difficulty and highlighting shortcomings in current VLMs. We additionally assess action recognition, sound recognition, and long-term video-object segmentation on HD-EPIC. HD-EPIC is 41 hours of video in 9 kitchens with digital twins of 413 kitchen fixtures, capturing 69 recipes, 59K fine-grained actions, 51K audio events, 20K object movements and 37K object masks lifted to 3D. On average, we have 263 annotations per minute of our unscripted videos. ![Image](images/aee6da957ed68e051c7414c79a9ba496430b69d8f44e530bb2e9de4d0f26caa3.jpg)
Figure 1. Annotation Highlights. We capture multi-day recordings of unscripted activities. Centre-Top: Recipes are recorded with steps and their preparation temporarily annotated, along with ingredient addition. Ingredients are weighed and nutrition recorded. Centre-Middle: Dense fine-grained narrations detailing what, how, and why are parsed and clustered. Audio events are also annotated. Centre-Bottom: Object movements are temporally annotated with bounding boxes and hands and object masks. Right-Top: All annotations are temporally grounded in a 3D digital twin. We show trajectories of 3 (masked) objects: Sweet potato, Food processor and Spoon, highlighting relevant kitchen fixtures. Right-Bottom: Gaze captures when objects are primed (i.e. looked at) before being taken/placed.
--- ### 1. Introduction Detailed understanding of videos, from the brief fine-grained action to the overarching hour-long activity, is effortless for humans but currently out of reach for both foundational and specialised models. Egocentric videos, in particular, introduce additional challenges to general video understanding, including significant camera motion, subtle action motion, objects occluded during manipulations and frequently going out of view. Understanding such videos requires disentangling the combined signals of head motion, hand interactions and a global understanding of the dynamic scene. This makes ego videos a great testbed for a comprehensive evaluation of video perception models. Egocentric vision has recently been fuelled by an influx of datasets [^18] [^28] [^29] [^80]. While large-scale, making them ideal for training, these datasets are sparsely annotated, particularly for tasks which link various parts of the long video, or those requiring 3D grounding. In contrast, richly annotated datasets tend to be synthetic or collected in controlled settings [^7] [^50] [^68] which limits their realism. We bridge this gap by presenting the most densely annotated dataset of unscripted recordings, ideal for comprehensive validation of video-only and video-language models. ★ Recipe steps are temporally annotated, and linked to annotations of all preparatory actions that relate to the step. We collect new videos, allowing us to capture additional meta-data and to ensure these videos have not already been used to train existing models. Following EPIC-Kitchens [^17], participants collect all kitchen activities for three days. We thus term our dataset Highly-Detailed EPIC (HD-EPIC). Fig. 1 provides an overview of the multi-tiered annotations, several of which are novel: ★ Each action has a dense description capturing the what, how, and why of actions along the start and end time. ★ Ingredients are weighed in videos and labelled with nutrition. We track dish nutrition as ingredients are added. ★ For each kitchen, we curate a digital twin with labelled fixtures. These are associated with actions (e.g. open/close) and the taking/placing of objects. ★ All moved objects are tracked, with manual masks lifted to 3D bounding boxes. $^{*}$ We associate gaze with object movements, labelling when objects are spotted before take/place actions. With these dense annotations, we design a challenging Visual Question Answering (VQA) benchmark of 26K questions. We purposefully do not use LLMs to generate negatives, instead using similar annotations. We highlight a few novel question types: ★ Recipe nutrition: we question the change in the recipe nutrition as one or more ingredients are added. $^{*}$ Multi-video: we question recipes prepared across recordings, with a VQA that spans multiple long videos. ★ Object itinerary: we question multi-hop object movements over a long video, relative to kitchen fixtures. ★ Fixture interactions: we question how many times a particular cupboard/drawer is opened/closed. ★ Action how/why: we question how/why an action was carried out, using participant-narrated manners/reasons. ★ Anticipation with gaze: With gaze priming, we query next-object movement, offering evidenced anticipation. Additionally, we report results on action recognition, sound recognition, and long-term video object segmentation. This paper thus contributes: (i) 41 hours of multi-day unscripted egocentric recordings, (ii) highly-detailed annotations including novel labels (e.g. ingredient nutrition, digital twin, gaze prime) and (iii) a challenging VQA benchmark including novel Qs (e.g. object itinerary, recipe nutrition changes) along with 3 standard video benchmarks. ### 2. Related Work With the rise of foundation models [^5] [^6] [^11] [^13] [^21] [^59] [^70] [^79] [^81] [^88], there has been a recent influx of benchmarks [8, 10, 15, 23, 25, 35, 42–44, 49, 58, 78, 92] aiming to test video understanding abilities. These benchmarks evaluate diverse capabilities e.g. physics [^58], counting [^23], temporal reasoning [^8] [^10] and long video [^23] [^25] [^49]. A few benchmarks test embodied or egocentric understanding. [^28] released a Natural Language Queries (NLQ) benchmark (19.2K queries) centred around episodic memory of objects. [^48] collects 1.6K human-made questions and answers on topics such as relative object locations, episodic memory, and spatial reasoning. However, it uses views from the HM3D [^63] and ScanNet [^16] datasets, so these questions are based on passive views of a static environment. [^49] [^86] auto-generate 5K and 7K questions based on Ego4D narrations. Whilst this approach is efficient, it is limited to these short narrations. [^8] collects its own annotations for videos from several datasets, including Ego4D. Their benchmark is solely focused on temporal questions related to ordering, counting, causality and direction. To evaluate a wider range of capabilities, a wider range of annotations are required. Of particular note are 3D grounding annotations. Ego4D [^28] contains some environment scans and static 3D object locations. With SLAM-equipped devices [^91] builds a benchmark for 3D object tracking; [^50] contains an office and living room digital twin; and [^29] contains ego- and exo- views of expert tasks. In contrast to these works which focus on only a few annotation types, we collect the most comprehensive set of annotations in one dataset, including highly detailed narrations, object and hand segmentations, and a comprehensive 3D digital twin of the scene and objects, all from unscripted egocentric footage in participants' homes. ### 3. Data Collection Recruitment and Equipment. Each participant engaged in a long commitment (~50 hours) involving data record- --- ![Image](images/3714815889c843526b65483dadad3ea4e20c97aaf8ae705028cfd918acfdd919.jpg)
Figure 2. Diversity in HD-EPIC, which is filmed over 3 days in-the-wild, resulting in many objects, activities and recipes.
![Image](images/321a5f79d554478c648b5c4ea250db2f9a2bfe3daa0bd5c93c2b9ddd4937e1f0.jpg)
Figure 3. Recipe modification in ingredients and steps.
ing and providing detailed narrations, recipes and nutrition information. Data was collected with Project Ariaglasses [^72]—a multi-sensor platform with 3 forward cameras (1 RGB and 2 SLAM), 7 microphones and inward cameras for gaze estimation. We collected 30 FPS RGB videos at $1408 \times 1408$ resolution, 60 FPS eye tracking and 30 FPS SLAM. We supplied participants with multiple devices including scales for nutritional tracking (see Fig. A1). Instructions and Collected Data. Participants recorded all their daily kitchen activities for at least 3 consecutive days. All 9 participants were asked to wear the glasses each time they walked into their kitchen, pressing record upon entering, and stopping the recording when they left the kitchen. Participants recorded for 3.5 to 7.2 hours (avg. 4.6). Overall, we collected 156 videos, with an average length of 15.9 ( $\pm$ 14.5) minutes totalling 41.3 hours (4.46M frames). Fig. 2 shows the diversity in the collected data. Following data collection, participants provided the recipes they freely prepared, citing the source (e.g. website) and any modifications (see Fig. 3). We collected a total of 69 recipes covering various cuisines. On average, recipes contained 6.6 steps, 8.1 ingredients, and took 4 hours 48 minutes across 2.1 videos from preparation to serving. Our longest recipe took 2 days and 6 hours to complete. To track nutrition of recipes, participants weighed and manually logged ingredients with MyFitnessPal [^3], giving us detailed nutrition information and adding an additional dimension to the dataset. In total, participants used 558 ingredients including ingredients high in protein, e.g. tuna and kidney beans; carbohydrates, e.g. dates and flour; and fat e.g. sour cream and pine nuts. Participants prepared both high calorie dishes e.g. Lazy Cake (4.8K calories) and low calorie dishes e.g. Crispy Cucumber Salad (274 calories). Narrations. We follow prior datasets [^17] [^18] [^28], asking participants to watch their recordings and narrate with a web-based narrator tool [^28]. We expand on this by asking participants to describe what they are doing, along with how and why. This results in a rich set of narrations that are denser, and more detailed than previous datasets (e.g. $3.8\times$ more words/min than Ego4D). See stats in Supp. B. Post-Processing—Multi-Video Slam and Gaze. We use Aria MPS [^1] to process videos obtaining singular multi-day point clouds per kitchen; 1kHz 6DoF camera trajectories; and eye gaze direction. We post-process VRS files, converting videos to mp4, removing the gaze camera input for anonymity. Further details are in Supp. B. ### 4. Annotation Pipeline We collect extensive multi-tiered annotations to achieve the level of detail that distinguishes HD-EPIC from other video understanding datasets. Here we detail our pipeline. #### 4.1. Annotating Recipe Steps and Ingredients Our videos are distinct from short recipe videos found online, which are typically trimmed to only crucial steps, and often edited further or sped up. Videos in HD-EPIC include a wider range of recipe-relevant activities, such as fetching or prepping ingredients. To comprehensively annotate these videos, we introduce prep and step pairs. The prep of a corresponding step is defined as all essential actions the participant takes to get ready to execute a given step. For example, the prep of the step ‘chop tomato’, includes retrieving the tomato from storage, washing it, and gathering the knife and chopping board. However, if the step is ‘Add chopped onions and stir’, then the chopping of onions is part of the prep for that step. This introduces a more fine-grained understanding of all steps, unexplored in prior datasets [^29] [^39] [^73]. Fig. 4 shows sample prep-step --- ![Image](images/8a426389e029a2387bc09bc091f5ceb21a5c8d0df15fc7bfe1677cfd537e8a68.jpg)
Figure 4. For the 'Carbonara' recipe, we visualise the prep and step time segments for three consecutive steps (left), along with sample frames with corresponding action narrations (top). The interleaving of different preps/steps is evident in the annotations.
![Image](images/13f0bfed169997cd1e1bf2bc3a3cbc3590589cd34f773b2ba81ed9baa442a4fe.jpg)
Figure 5. Frequency of verb clusters (top) and noun clusters (bottom) in narrated sentences by category, shown on a logarithmic scale.
![Image](images/9862df2132efd6f2108c23ed93c72ebcf5ad279cc0aa114716778927e3b348d9.jpg)
Figure 6. Nutrition is monitored throughout recipes as ingredients are incorporated into dishes. Here we show Banana Bread Chocolate Chip Cookies. We annotate when ingredients are weighed, document their nutrition, and locate their adding time, giving us overall dish nutrition at each stage.
annotations for 3 steps. Nearly all steps (93.1%) have paired prep annotations. Typically, prep is shorter than a step: avg. prep is 54.5s ( $\pm$95.3s), avg. step is 78.2s ( $\pm$100.7s). We also annotate weighing and adding temporal segments which enables monitoring the nutrition of the full dish as ingredients are incorporated (see Fig. 6). In total, we annotate 283 in-view weighing sequences (avg. 18.9s) and 501 adding sequences (avg. 31.6s), excluding spices. Details of the annotation process are in Supp. C. #### 4.2. Fine-Grained Actions Transcription. We automatically transcribe and manually check and correct all audio narrations provided by participants, to obtain detailed action descriptions. Action Boundaries. For all narrations, we label precise start and end times. In total, we obtain segments for 59,454 actions, with a mean duration of 2.0s ( $\pm$3.4s). Parsing. We parse verbs, nouns and hands from open vocabulary narrations so they can be used for closed vocabulary tasks, such as action recognition. We also extract how and why clauses from 16,004 and 11,540 narrations, respectively. For example, “Turn the salt container clockwise by pushing it with my left hand so that the lid is aligned with the container opening.” Clustering. Fig. 5 shows the distribution of clusters (i.e. classes) across all videos in HD-EPIC, along with hierarchical clusters [^18]. As with prior datasets [^18] [^28], our highly diverse actions and objects are long-tailed. Sound Annotations. We follow [^31] to collect audio annotations. These capture start-end times of audio events along with a class name (e.g. 'click', 'rustle', 'metal-plastic collision', 'running water'). Overall, we have 50,968 audio annotations from 44 classes. Full details of transcription, boundary labelling, parsing, clustering and sound annotationss are in Supp. C. --- ![Image](images/d6a83e877fb791332de355ae4b09c427f44a18ffe772913abbda36b0b403e836.jpg)
Figure 7. Digital Twin: from point cloud (left), to surfaces (middle) and labelled fixtures (right). We show two moved objects (masks on top) at fixtures: cheese and pan. Body poses from [^87].
#### 4.3. Digital Twins: Scene & Object Movements Scene. We create digital copies of participants' kitchens by reconstructing the surfaces and manually curating every fixture (e.g. cupboard, drawer), storage space (e.g. shelves, hooks) and large appliance (e.g. fridge, microwave). This is distinct from digital twins that rely on known environments with replicas. Our digital twin is created in Blender [^2] on top of the multi-video SLAM point clouds from recordings. Each kitchen contains an average of 45.9 labelled fixtures (min 31, max 62), including 14.2 counters/surfaces, 12.2 cupboards, 7.8 drawers and 5.2 appliances (sample in Fig. 7). We refer to these annotations as Fixtures F. We then associate narrations which describe scene interactions with F. We find actions where a noun indicates a fixture, e.g. “open drawer”, identify the exact “drawer” in the digital twin (e.g. drawer.001) and update its state. Following studies showing humans fixate up to 1 second before interacting [^40], we take the fixture $f \in F$ with the highest cumulative gaze intersection for the 1s before the narration. \textbf{Hand Masks.} We annotate a handful of frames per video for both hands. Frames are selected to cover various actions and kitchen locations. We use these to automatically segment, and manually correct a selected subset. In total, our dataset contains 7.7M hand masks: 3.9M right and 3.8M left of which 11K are manually annotated (details in Supp. C). Moving Objects in 2D. To generate 3D object movement annotations, we first annotate when objects move. Annotators label a temporal segment each time an object is moved until it is set, along with 2D object bounding boxes at the onset and end of motion. For example, if a person moves a cup from a countertop to the sink, one bounding box captures the cup on the countertop and another when in the sink. Tracks are annotated even for slight shifts/pushes, and ![Image](images/913b85b351097a9d16fbc8049e1f9b96846556bdeb11b7954a2321a27987ce2f.jpg)
Figure 8. Priming Object Interaction Through Gaze. Top: Camera position with projected eye-gaze and object positions in 3D. Middle: 2D gaze location. Bottom: Timeline for priming object movement e.g. the glass is primed 8.3s before taking.
thus offer full annotations of all object movements. Overall, we collected 19.9K object movement tracks and 36.9K bounding boxes. We label an average of 9.2 objects taken and 9.0 objects placed per minute. On average, tracks are 9.0s long, the longest is 461.5s. Object Masks. Despite progress in segmentation [^36] [^64] and available annotations [^19] [^29], models perform poorly in egocentric video, particularly under occlusions. We obtain pixel-level segmentations from each bounding box by initialising with iterative SAM2 [^64] then manually correcting. Annotators corrected 74% of masks; the IoU between SAM2 and the manual masks is 0.82. Masks to 3D. We lift object masks to 3D using dense depth estimates and 2D-to-3D sparse correspondences provided by MPS. Given metric depth from [^85], we identify S, the set of pixels within or around the object with 3D correspondences. We then find the linear transformation coefficients: $\alpha, \beta = \arg\min_{\alpha, \beta} \|\left(\alpha\hat{D}_S + \beta\right) - D_S\|^2$. where $\hat{D}_S$ are estimated depth values and $D_S$ are existing depth values, followed by RANSAC to remove outliers. 3D Object Motion. Objects move 61.4cm (±84.5cm) on average, 27.6% move ≤10cm, while 7.6% move ≥2m. Object-Scene Interactions. With the 3D object locations, we associate locations with the closest fixture $f \in F$, subject to fixture-specific heuristics (e.g. objects must be within a counter's x-y plane). We manually verify all assignments, correcting any errors. On average objects move between 1.8 different fixtures per video (see Supp. C for stats). Priming Object Movement. The behaviour of gaze when picking up and placing objects is well-studied [^32] [^40]. We combine eye-gaze and 3D object locations, to find when an object is primed, i.e. the moment in time when the gaze attends to the object's location before picking it up (pick-up priming) or when the gaze attends to the future location of an object before it's put down (put-down priming). We calculate the priming time for all objects, excluding those taking or placed off screen. Additionally, at times, a person is already manipulating an object well before picking it up. We thus exclude objects with a pick up location. ---
DatasetVal&Test HoursAction SegmentsUnscriptedFree SettingRecipeNutritionGazeAudio LabelsObjectHand3D object over timeLabelled 3D environmentCamera poseFully annotated
HOI4D [^46]11.4✓✗✗✗✗✗✗MaskMask✓✓✗✓
Assembly101 [^68]66.8✓✗✗✗✗✗✗✗3D pose✗✗✗✓
EPIC-KITCHENS-100 [^18]25.3✓✓✓✗✗✗✓MaskMask✓✗✓✗
Ego4D [^28]288.7✓✓✓✗✗✗✗B-BoxB-Box✗✗✗✗
HoloAssist [^80]49.8✓✗✗✗✗✗✓✗3D pose✗✗✓✓
Aria Digital Twin [^50]8.1✗✗✗✗✗✓✗Mask✗✓✓✓✓
Aria Everyday Activities [^47]7.3✗✗✓✗✗✓✗✗✗✗✗✓✗
Aria Everyday Objects [^74]0.4✗✓✓✗✗✗✗B-Box✗✓✗✓✓
Ego-Exo4D [^29]85.1✗✓✓✓✗✓✗Mask3D pose✗✗✓✗
HD-EPIC41.3✓✓✓✓✓✓✓MaskMask✓✓✓✓
Table 1. Comparison of Egocentric Video Datasets (see full table A1 in Supp.).
LocationFiltered (% Total)Feasible (% Total)Primed (% Feasible)Not Primed (% Feasible)Avg. Time (s)
Start29.4070.6094.825.183.99( $\pm$2.94)
End66.9233.0888.4611.542.62( $\pm$2.05)
![Image](images/6af34033266051fab972eaea598530a0f86df0f93c2b3eb156a703cbe32a02a7.jpg)
Figure 9. (Top) Priming Statistics for both start and end locations (Bottom) Histogram showing the difference in time when an object is primed before it is picked up (blue) or placed (red).
already close to the gaze 10s earlier. In Fig. 8 we show gaze priming for two objects: milk bottle and glass. The glass's end location, a cupboard, is primed 3s before the glass is put away. Fig. 9 displays priming statistics. Of those objects feasible for priming, 94.8% are primed, an average of 4.0s before being picked up, compared to 88.5% primed an average of 2.6s before being placed. Long Term Object Tracking. We connect object movements and form longer trajectories, i.e. object itineraries, to capture sequences of an object's movement. Our efficient pipeline utilises our lifted 3D locations and allows annotating a 1-hour long video in minutes (details in Supp. C). #### 4.4. HD-EPIC vs Prior Egocentric Datasets Tab. 1 compares HD-EPIC to other egocentric datasets (full table in Supp.). Compared to the largest dataset with labelled 3D environments (Aria Digital Twin [^50]), HD-EPIC contains 5x more footage; has more annotations; and importantly was collected in an unscripted manner in the participants' homes. In particular, HD-EPIC is the first to annotate recipes, nutritional values, detailed action segments, gaze and audio labels on the same set of videos. With these diverse and dense annotations, HD-EPIC constitutes a true zero-shot benchmark for video understanding. ### 5. Benchmarks and Results We show the potential of HD-EPIC as a validation dataset with benchmarks on general Video Question Answering (VQA) (Sec 5.1), action and sound recognition (Sec 5.2) and long-term video object segmentation (Sec 5.3). ![Image](images/d976114fa4252ca0ee33a3e9dc530129456a0b63e613d92a7606dfd6fd48e6e4.jpg)
Figure 10. VQA Question Prototypes. We show our 30 question prototypes by category alongside the number of questions. Outer bars indicate the distribution over input lengths for each question.
#### 5.1. HD-EPIC VQA Benchmark and Analysis Benchmark Creation. We take the dense output of our annotation pipeline and construct a comprehensive VQA benchmark around 7 types of annotations: 1. Recipe. Questions on temporally localising, retrieving, or recognising recipes and their steps. 2. Ingredient. Questions on the ingredients used, their weight, their adding time and order. 3. Nutrition. Questions on nutrition of ingredients and nutritional changes as ingredients are added to recipes. 4. Fine-grained action. What, how, and why of actions and their temporal localisation. 5. 3D perception. Questions that require the understanding of relative positions of objects in the 3D scene. 6. Object motion. Questions on where, when and how many times objects are moved across long videos. 7. Gaze. Questions on estimating the fixation on large landmarks and anticipating future object interactions. For each question type, we define prototypes to sample questions, correct answers, and strong negatives from our annotations. For example, Object Movement Counting asks "How many times did the object seen at