Second Joint Egocentric Vision Workshop (original) (raw)

Overview

Wearable cameras, smart glasses, and AR/VR headsets are gaining importance for research and commercial use. They feature various sensors like cameras, depth sensors, microphones, IMUs, and GPS. Advances in machine perception enable precise user localization (SLAM), eye tracking, and hand tracking. This data allows understanding user behavior, unlocking new interaction possibilities with augmented reality. Egocentric devices may soon automatically recognize user actions, surroundings, gestures, and social relationships. These devices have broad applications in assistive technology, education, fitness, entertainment, gaming, eldercare, robotics, and augmented reality, positively impacting society.

Previously, research in this field faced challenges due to limited datasets in a data-intensive environment. However, the community's recent efforts have addressed this issue by releasing numerous large-scale datasets covering various aspects of egocentric perception, including HoloAssist, Ego4D, Ego-Exo4D, EPIC-KITCHENS, and HD-EPIC.

The goal of this workshop is to provide an exciting discussion forum for researchers working in this challenging and fast-growing area, and to provide a means to unlock the potential of data-driven research with our datasets to further the state-of-the-art.

Challenges

We welcome submissions to the challenges from February to May (see important dates) through the leaderboards linked below. Participants to the challenges are requested to submit a technical report on their method. This is a requirement for the competition. Reports should be 2-6 pages including references. Submissions should use the CVPR format and should be submitted through the CMT website.

HoloAssist Challenges

HoloAssist is a large-scale egocentric human interaction dataset, where two people collaboratively complete physical manipulation tasks.

Action Recognition

**Lead:**Taein Kwon, ETH Zurich, Switzerland
**Summary:**Action Recognition on the holoassist dataset. Input could be RGB images or multiple modalities.

Challenge Link

Mistake Detection

**Lead:**Mahdi Rad, Microsoft, Switzerland
**Summary:**Mistake detection is defined following the convention Assembly101 but applied to fine-grained actions in our benchmark. We take the features from the fine-grained action clips from the beginning of the coarse-grained action until the end of the current action clip, and the model predicts a label from {correct, mistake}.

Challenge Link

Ego4D Challenges

Ego4D is a massive-scale, egocentric dataset and benchmark suite collected across 74 worldwide locations and 9 countries, with over 3,670 hours of daily-life activity video. Please find details below on our challenges:

Ego4D Episodic Memory

Track: Visual Queries

Lead: Suyog Jain, Meta, US
**Summary:**Given an egocentric video, the goal is to answer queries of the form "Where did I last see object X?", where the query object X is specified as a static image.

Challenge Link

Ego4D Episodic Memory

Track: Natural Language Queries

Lead: Suyog Jain, Meta, US
**Summary:**Given an egocentric video V and a natural language query Q, the goal is to identify a response track r, such that the answer to Q can be deduced from r.

Challenge Link

Ego4D Episodic Memory

Track: Moment Queries

**Lead:**Chen Zhao & Merey Ramazanova, KAUST, SA
Summary: Given an input video and a query action category, the goal is to retrieve all the instances of this action category in the video.
Current SOTA: Paper
Previous Winner: 34.99

Challenge Link

Ego4D Episodic Memory

Track: Goal Step

Lead: Yale Song, Meta, US
Summary: Given an untrimmed egocentric video, identify the temporal action segment corresponding to a natural language description of the step. Specifically, predict the (start_time, end_time) for a given keystep description.
Current SOTA: Paper
Previous Winner: 35.18 r@1, IoU=0.3

Challenge Link

Ego4D Episodic Memory

Track: EgoSchema

Lead: Karttikeya Mangalam & Raiymbek Akshulakov, UC Berkeley, US
Summary: EgoSchema is a very long-form video question-answering dataset and benchmark to evaluate long video understanding capabilities of modern vision and language systems.
Current SOTA: 0.75 (report unavailable)
Previous Winner: N/A

Challenge Link

Ego4D Social Interaction

Track: Looking at me

Lead: Xizi Wang, Indiana University, US
**Summary:**The task focuses on identifying communicative acts that are directed towards the camera-wearer, as distinguished from those directed to other social partners

Challenge Link

Ego4D Social Interaction

Track: Talking to me

**Lead:**Xizi Wang, Indiana University, US
**Summary:**Given a video and audio segment with the same tracked faces and an additional label that identifies speaker status, classify whether each visible face is talking to the camera wearer.

Challenge Link

Ego4D Forecasting

Track: Short-term object interaction anticipation

**Lead:**Francesco Ragusa, University of Catania, IT
**Summary:**This task aims to predict the next human-object interaction happening after a given timestamp. Given an input video, the goal is to anticipate 1)the spatial positions of the active objects, 2) the category of each detected next active objects, 3) how each active object will be used (verb), 4) and when the interaction will begin.
Current SOTA:Paper 1; Paper 2
Previous Winner: Top-5 Overall mAP: 7.21

Challenge Link

Ego4D Forecasting

Track: Long-term action anticipation

Lead: Tushar Nagarajan, FAIR, US
Summary: This task aims to predict the next Z future actions after a given action. Given an input video up to a particular timestep (corresponding to the last visible action), the goal is to predict a list of action classes [(verb1, noun1), (verb2, noun2) ... (verbZ, nounZ)] that follow it.
Current SOTA: Paper
Previous Winner: N/A

Challenge Link

Ego-Exo4D Challenges

Ego-Exo4D is a diverse, large-scale multi-modal multi view video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured ego- centric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair).

EgoExo4D Pose Challenge

Track: Ego-Pose Body

Lead: Juanita Puentes Mozo & Gabriel Perez Santamaria, Los Andes
Summary: The EgoExo4D Body Pose Challenge aims to accurately estimate body pose using only first-person raw video and/or egocentric camera pose.
Current SOTA: EgoCast (MPJPE: 14.36)
Previous Winner: MPJPE: 15.32

Challenge Link

EgoExo4D Pose Challenge

Track: Ego-Pose Hands

**Lead:**Shan Shu, University of Pennsylvania, US
Summary:
Current SOTA:
Previous Winner:

Challenge Link

EgoExo4D Proficiency Estimation

Track: Demonstrator Proficiency

Lead: Arjun Somayazulu, UT Austin, US
Summary: Given synchronized egocentric and exocentric video of a demonstrator performing a task, classify the proficiency skill level of the demonstrator.
Current SOTA: EgoExo4D benchmark baseline
Previous Winner: N/A

Challenge Link

EgoExo4D Keysteps

Track: Fine-grained Keystep Recognition

Lead: Sherry Xue & Tushar Nagarajan, UT Austin, US
Summary:
Current SOTA:
Previous Winner:

Challenge Link

EgoExo4D Relations

Track: Correspondence

Lead: Sanjay Haresh, Simon Fraser, Canada
Summary: The challenge is aimed at methods for object correspondences across ego-centric and exo-centric views. Given a pair of time-synchronized egocentric and exocentric videos, as well as a query object track in one of the views, the goal is to output the corresponding mask for the same object instance in the other view for all frames where the object is visible in both views.
Current SOTA: Paper
Previous Winner: N/A

Challenge Link

EgoExo4D Keysteps

Track: Procedure Understanding

**Lead:**Antonino Furnari, University of Catania, IT
**Summary:**The objective of this task is to infer a procedure's underlying structure from observing natural videos of subjects performing the procedure.
Current SOTA:
Previous Winner:

Challenge Link

EPIC-Kitchens Challenges

Please check the EPIC-KITCHENS website for more information on the EPIC-KITCHENS challenges. Links to individual challenges are also reported below.

Action Recognition

Lead: Prajwal Gatti and Siddhant Bansal, University of Bristol, UK
Summary: Classify the action's verb and noun depicted in a trimmed video clip.
Current SOTA: Paper
Previous Winner: 48.1% - top 1 / 77.4% - top 5

Challenge Link

Action Detection

Lead: Francesco Ragusa and Antonino Furnari, University of Catania, IT
Summary: The challenge requires detecting and recognising all action instances within an untrimmed video. The challenge will be carried out on the EPIC-KITCHENS-100 dataset.
Current SOTA: Results
Previous Winner: Action Avg. mAP 31.97

Challenge Link

Domain Adaptation Challenge for Action Recognition

Lead: Saptarshi Sinha and Prajwal Gatti, University of Bristol, UK
Summary: Given labelled videos from the source domain and unlabelled videos from the target domain, the goal is to classify actions in the target domain. An action is defined as a verb and noun depicted in a trimmed video clip.
Current SOTA: Paper
Previous Winner: 43.17 for action accuracy

Challenge Link

Multi-Instance Retrieval

Lead: Prajwal Gatti and Michael Wray, University of Bristol, UK
Summary: Perform cross-modal retrieval by searching between vision and text modalities.
Current SOTA: Paper
Previous Winner: Normalised Discounted Cumulative Gain (%) Avg. - 74.25

Challenge Link

Semi-Supervised Video-Object Segmentation

Lead: Rhodri Guerrier and Ahmad Darkhalil, University of Bristol, UK
**Summary:**Given a sub-sequence of frames with M object masks in the first frame, the goal of this challenge is to segment these through the remaining frames. Other objects not present in the first frame of the sub-sequence are excluded from this benchmark.
Current SOTA:Webpage

Challenge Link

EPIC-SOUNDS Audio-Based Interaction Recognition

Lead: Omar Emara and Jacob Chalk, University of Bristol, UK
Summary: Recognising interactions from audio data from EPIC-Sounds (classify the audio).
Current SOTA: User: JMCarrot
Previous Winner: N/A

Challenge Link

EPIC-SOUNDS Audio-Based Interaction Detection

Lead: Omar Emara and Jacob Chalk, University of Bristol, UK
Summary: Classify all audio-based interactions (recognition) from audio data of EPIC-Sounds and predict their start and end times for a given video.
Current SOTA: User: shuming
Previous Winner: N/A

Challenge Link

HD-EPIC Challenge

Please check the HD-EPIC website for more information on the HD-EPIC challenges. Links to individual challenges are also reported below.

HD-EPIC Challenges - VQA

Lead: Prajwal Gatti (University of Bristol, UK) and Kaiting Liu (Leiden University, Netherlands)
Summary: Given a question belonging to any one of the seven types defined in the HD-EPIC VQA benchmark, the goal is to predict the correct answer among the five listed choices.
Current SOTA: Gemini Pro
Previous Winner: N/A

Challenge Link

Call for Abstracts

You are invited to submit extended abstracts to the second edition of joint egocentric vision workshop which will be held alongside CVPR 2025 in Nashville.

These abstracts represent existing or ongoing work and will not be published as part of any proceedings. We welcome all works that focus within the Egocentric Domain, it is not necessary to use the Ego4D dataset within your work. We expect a submission may contain one or more of the following topics (this is a non-exhaustive list):

• Egocentric vision for human activity analysis and understanding, including action recognition, action detection, audio-visual action perception and object state change detection
• Egocentric vision for anticipating human behaviour, actions and objects
• Egocentric vision for 3D perception and interaction, including dynamic scene reconstruction, hand-object reconstruction, long-term object tracking, NLQ and visual queries, long-term video understanding
• Head-mounted eye tracking and gaze estimation including attention modelling and next fixation prediction
• Egocentric vision for object/event recognition and retrieval
• Egocentric vision for summarization
• Daily life and activity monitoring
• Egocentric vision for human skill learning, assistance, and robotics
• Egocentric vision for social interaction and human behaviour understanding
• Privacy and ethical concerns with wearable sensors and egocentric vision
• Egocentric vision for health and social good
• Symbiotic human-machine vision systems, human-wearable devices interaction
• Interactive AR/VR and Egocentric online/real-time perception

Format

The length of the extended abstracts is 2-4 pages, including figures, tables, and references. We invite submissions of ongoing or already published work, as well as reports on demonstrations and prototypes. The joint egocentric vision workshop gives opportunities for authors to present their work to the egocentric community to provoke discussion and feedback. Accepted work will be presented as either an oral presentation (either virtual or in-person) or as a poster presentation. The review will be single-blind, so there is no need to anonymize your work, but otherwise will follow the format of the CVPR submissions, information can be foundhere. Accepted abstracts will not be published as part of a proceedings, so can be uploaded to ArXiv etc. and the links will be provided on the workshop’s webpage. The submission will be managed with the CMT website.

Important Dates

Program

All dates are local to Nashville's time, CST.
Workshop Location: Room Grand B1

Papers

Note to authors: Please hang your poster in Hall D against the boards with workshop's name. Posters can be put up ONLY during the poster session time (10.00 - 11.00).
Follow CVPR poster guidelines for poster dimensions.

All workshop posters are in Exhibition Hall D.

Extended Abstracts

Invited CVPR Papers

Invited Speakers

Workshop Organisers

Co-organizing Advisors

Related Past Events

This workshop follows the footsteps of the following previous events:

• EgoVis@CVPR2024: First Joint Egocentric Vision (EgoVis) Workshop in conjunction with CVPR 2024;

EPIC-Kitchens and Ego4D Past Workshops:

• Ego4D&EPIC@CVPR2023: Joint International 1st Ego4D and 10th EPIC Workshop in conjunction with CVPR 2023;
• 2nd International Ego4D Workshop in conjunction with ECCV 2022;
• Ego4D&EPIC@CVPR2022: Joint International 1st Ego4D and 10th EPIC Workshop in conjunction with CVPR 2022;
• EPIC@ICCV21: The Ninth International Workshop on Egocentric Perception, Interaction and Computing in conjunction with ICCV 2021;
• EPIC@CVPR21: The Eighth International Workshop on Egocentric Perception, Interaction and Computing in conjunction with CVPR 2021;
• EPIC@ECCV20: The Seventh International Workshop on Egocentric Perception, Interaction and Computing in conjunction with ECCV 2020;
• EPIC@CVPR20: The Sixth International Workshop on Egocentric Perception, Interaction and Computing in conjunction with CVPR 2020;
• EPIC@ICCV19: The Fifth International Workshop on Egocentric Perception, Interaction and Computing in conjunction with ICCV 2019;
• EPIC@CVPR19: The Fourth International Workshop on Egocentric Perception, Interaction and Computing in conjunction with CVPR 2019;
• EPIC@ECCV18: The Third International Workshop on Egocentric Perception, Interaction and Computing in conjunction with ECCV 2018;
• EPIC@ICCV17: The Second International Workshop on Egocentric Perception, Interaction and Computing in conjunction with ICCV 2017;
• EPIC@ECCV16: The First International Workshop on Egocentric Perception, Interaction and Computing in conjunction with ECCV 2016;

Human Body, Hands, and Activities from Egocentric and Multi-view Cameras Past Workshops:

• Human Body, Hands, and Activities from Egocentric and Multi-view Cameras (HBHA) run alongside ECCV 2022;

Project Aria Past Tutorials:

• Aria tutorial run alongside CVPR 2023;
• Aria tutorial run alongside CVPR 2022;