Probabilistic Detection of Pointing Directions for Human-Robot Interaction (original) (raw)

Visual recognition of pointing gestures for human–robot interaction

Image and Vision Computing, 2007

In this paper, we present an approach for recognizing pointing gestures in the context of human-robot interaction. In order to obtain input features for gesture recognition, we perform visual tracking of head, hands and head orientation. Given the images provided by a calibrated stereo camera, color and disparity information are integrated into a multi-hypothesis tracking framework in order to find the 3D-positions of the respective body parts. Based on the hands' motion, an HMM-based classifier is trained to detect pointing gestures. We show experimentally that the gesture recognition performance can be improved significantly by using information about head orientation as an additional feature. Our system aims at applications in the field of human-robot interaction, where it is important to do run-on recognition in real-time, to allow for robot egomotion and not to rely on manual initialization.

Real-time recognition of pointing gestures for robot to robot interaction

2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014

This paper addresses the idea of establishing symbolic communication between mobile robots through gesturing. Humans communicate using body language and gestures in addition to other linguistic modalities like prosody and text or dialog structure. This research aims to develop a pointing gesture detection system for robot to robot communication scenarios to grant robots an ability to convey object identity information without global localization of the agents. The detection is based on RGB-D and a NAO humanoid robot is used as the pointing agent in the experiments. The presented algorithms are based on PCL library. The results indicate that real-time detection of pointing gesture can be performed with little information about the embodiment of the pointing agent and that an observing agent can use the gesture detection to perform actions on the pointed targets.

Precise pointing target recognition for human-robot interaction

This work presents a person independent pointing gesture recognition application. It uses simple but effective features for the robust tracking of the head and the hand of the user in an undefined environment. The application is able to detect if the tracking is lost and can be reinitialized automatically. The pointing gesture recognition accuracy is improved by the proposed fingertip detection algorithm and by the detection of the width of the face. The experimental evaluation with eight different subjects shows that the overall average pointing gesture recognition rate of the system for distances up to 250 cm (head to pointing target) is 86.63% (with a distance between objects of 23 cm). Considering just frontal pointing gestures for distances up to 250 cm the gesture recognition rate is 90.97% and for distances up to 194 cm even 95.31%. The average error angle is 7.28 • .

Hand Gesture Pointing Location Detection

2014

Gesture recognition has been implemented in many systems for different purposes. The way it is used can be different from system to system. Few used them as touch objects using sensors while few use them as pointing object where the camera is in front of the hand, but the location is being pointed by human can be detected only if the camera is behind that person. This paper presents an automatic system to detect the location on the wall where the user is currently pointing by his hand. This is done with the detection of head and both hands. The shoulders and elbows' locations are identified using geometry analysis to understand the current gesture of the human and finally the direction where user hand is pointing would be detected. The face was detected using Haar classifier and shoulders were estimated using the rectangle method near the head. This method does not work in poor illumination conditions and is made to detect only one person at a time. Also user is supposed to wear half sleeve or sleeveless shirt for the better segmentation. the input domain, the direct sensing approach requires capturing and interpreting the motion of head, eye gaze, face, hand, arms or even the whole body. This paper aims at finding the location of palms, shoulders and elbows. By doing this we will be aim to classify what is being pointed at.

3D pointing gesture recognition for human-robot interaction

2016 Chinese Control and Decision Conference (CCDC), 2016

In this paper, a pointing gesture recognition method is proposed for human-robot interaction. The pointing direction of the human partner is obtained by extracting the joint coordinates and computing through vector calculations. 3D to 2D mapping is implemented to build a top-view 2D map with respect to the actual ground circumstance. Using this method, robot is able to interpret the human partner's 3D pointing gesture based on the coordinate information of his/her shoulder and hand. Besides this, speed control of robot can be achieved by adjusting the position of the human partner's hand relative to the head. The recognition performance and viability of the system are tested through quantitative experiments.

Recognition and Estimation of Human Finger Pointing with an RGB Camera for Robot Directive

arXiv (Cornell University), 2023

In communication between humans, gestures are often preferred or complementary to verbal expression since the former offers better spatial referral. Finger pointing gesture conveys vital information regarding some point of interest in the environment. In human-robot interaction, a user can easily direct a robot to a target location, for example, in search and rescue or factory assistance. State-of-the-art approaches for visual pointing estimation often rely on depth cameras, are limited to indoor environments and provide discrete predictions between limited targets. In this paper, we explore the learning of models for robots to understand pointing directives in various indoor and outdoor environments solely based on a single RGB camera. A novel framework is proposed which includes a designated model termed PointingNet. PointingNet recognizes the occurrence of pointing followed by approximating the position and direction of the index finger. The model relies on a novel segmentation model for masking any lifted arm. While state-of-the-art human pose estimation models provide poor pointing angle estimation accuracy of 28 • , PointingNet exhibits mean accuracy of less than 2 •. With the pointing information, the target is computed followed by planning and motion of the robot. The framework is evaluated on two robotic systems yielding accurate target reaching.

Robotic System Capable of Identifying Objects Indicated by Pointing Gestures

This work presents a robotic system able to visually segment an unknown object that was indicated by a human through a pointing gesture. The robot uses RGB-D sensors to observe the human and find the 3D point indicated by the pointing gesture. The system can use this point to initialize a fixationbased, fast object segmentation algorithm, inferring thus the outline of the whole object. A series of experiments with different objects and pointing gestures show that both the recognition of the gesture, the extraction of the pointing direction in 3D, and the object segmentation perform robustly. The discussed system can provide the first step towards more complex tasks, such as object recognition, grasping or learning by demonstration with obvious value in both industrial and domestic settings.

There You Go! - Estimating Pointing Gestures In Monocular Images For Mobile Robot Instruction

ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication, 2006

In this paper, we present a neural architecture that is capable of estimating a target point from a pointing gesture, thus enabling a user to command a mobile robot to a specific position in his local surroundings by means of pointing. In this context, we were especially interested to determine whether it is possible to implement a target point estimator using only monocular images of low-cost webcams. The feature extraction is also quite straightforward: We use a gabor jet to extract the feature vector from the normalized camera images; and a cascade of Multi Layer Perceptron (MLP) Classifiers as estimator. The System was implemented and tested on our mobile robotic assistant HOROS. The results indicate that it is in fact possible to realize a pointing estimator using monocular image data, but further efforts are necessary to improve the accuracy and robustness of our approach.

A multi-view hand gesture RGB-D dataset for human-robot interaction scenarios

2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2016

Understanding semantic meaning from hand gestures is a challenging but essential task in human-robot interaction scenarios. In this paper we present a baseline evaluation of the Innsbruck Multi-View Hand Gesture (IMHG) dataset [1] recorded with two RGB-D cameras (Kinect). As a baseline, we adopt a probabilistic appearance-based framework [2] to detect a hand gesture and estimate its pose using two cameras. The dataset consists of two types of deictic gestures with the ground truth location of the target, two symbolic gestures, two manipulative gestures, and two interactional gestures. We discuss the effect of parallax due to the offset between head and hand while performing deictic gestures. Furthermore, we evaluate the proposed framework to estimate the potential referents on the Innsbruck Pointing at Objects (IPO) dataset [2].