Exploring the Potential of Combining Time of Flight and Thermal Infrared Cameras for Person Detection (original) (raw)
Related papers
Towards people detection from fused time-of-flight and thermal infrared images
ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2014
Obtaining accurate 3d descriptions in the thermal infrared (TIR) is a quite challenging task due to the low geometric resolutions of TIR cameras and the low number of strong features in TIR images. Combining the radiometric information of the thermal infrared with 3d data from another sensor is able to overcome most of the limitations in the 3d geometric accuracy. In case of dynamic scenes with moving objects or a moving sensor system, a combination with RGB cameras of Time-of-Flight (TOF) cameras is suitable. As a TOF camera is an active sensor in the near infrared (NIR) and the thermal infrared camera captures the radiation emitted by the objects in the observed scene, the combination of these two sensors for close range applications is independent from external illumination or textures in the scene. This article is focused on the fusion of data acquired both with a time-of-flight (TOF) camera and a thermal infrared (TIR) camera. As the radiometric behaviour of many objects differs between the near infrared used by the TOF camera and the thermal infrared spectrum, a direct co-registration with feature points in both intensity images leads to a high number of outliers. A fully automatic workflow of the geometric calibration of both cameras and the relative orientation of the camera system with one calibration pattern usable for both spectral bands is presented. Based on the relative orientation, a fusion of the TOF depth image and the TIR image is used for scene segmentation and people detection. An adaptive histogram based depth level segmentation of the 3d point cloud is combined with a thermal intensity based segmentation. The feasibility of the proposed method is demonstrated in an experimental setup with different geometric and radiometric influences that show the benefit of the combination of TOF intensity and depth images and thermal infrared images.
Fusion of Range and Thermal Images for Person Detection
Detecting people in images is a challenging problem. Differences in pose, clothing and lighting, along with other factors, cause a lot of variation in their appearance. To overcome these issues, we propose a system based on fused range and thermal infrared images. These measurements show considerably less variation and provide more meaningful information. We provide a brief introduction to the sensor technology used and propose a calibration method. Several data fusion algorithms are compared and their performance is assessed on a simulated data set. The results of initial experiments on real data are analysed and the measurement errors and the challenges they present are discussed. The resulting fused data are used to efficiently detect people in a fixed camera set-up. The system is extended to include person tracking.
Robust Pedestrian Detection by Combining Visible and Thermal Infrared Cameras
Sensors, 2015
With the development of intelligent surveillance systems, the need for accurate detection of pedestrians by cameras has increased. However, most of the previous studies use a single camera system, either a visible light or thermal camera, and their performances are affected by various factors such as shadow, illumination change, occlusion, and higher background temperatures. To overcome these problems, we propose a new method of detecting pedestrians using a dual camera system that combines visible light and thermal cameras, which are robust in various outdoor environments such as mornings, afternoons, night and rainy days. Our research is novel, compared to previous works, in the following four ways: First, we implement the dual camera system where the axes of visible light and thermal cameras are parallel in the horizontal direction. We obtain a geometric transform matrix that represents the relationship between these two camera axes. Second, two background images for visible light and thermal cameras are adaptively updated based on the pixel difference between an input thermal and pre-stored thermal background images. Third, by background subtraction of thermal image considering the temperature characteristics of background and size filtering with morphological operation, the candidates from whole image (CWI) in the thermal image is obtained. The positions of CWI (obtained by background subtraction and the procedures of shadow removal, morphological operation, size filtering, and filtering of the ratio of height to width) in the visible light image are
Improving Person Tracking Using an Inexpensive Thermal Infrared Sensor
2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014
This paper proposes a person tracking framework using a scanning low-resolution thermal infrared (IR) sensor colocated with a wide-angle RGB camera. The low temporal and spatial resolution of the low-cost IR sensor make it unable to track moving people and prone to false detections of stationary people. Thus, IR-only tracking using only this sensor would be quite problematic. We demonstrate that despite the limited capabilities of this low-cost IR sensor, it can be used effectively to correct the errors of a real-time RGB camera-based tracker. We align the signals from the two sensors both spatially (by computing a pixelto-pixel geometric correspondence between the two modalities) and temporally (by modeling the temporal dynamics of the scanning IR sensor), which enables multi-modal improvements based on judicious application of elementary reasoning. Our combined RGB+IR system improves upon the RGB camera-only tracking by: rejecting false positives, improving segmentation of tracked objects, and correcting false negatives (starting new tracks for people that were missed by the camera-only tracker). Since we combine RGB and thermal information at the level of RGB camera-based tracks, our method is not limited to the particular camerabased tracker that we used in our experiments. Our method could improve the results of any tracker that uses RGB camera input alone. We collect a new dataset and demonstrate the superiority of our method over RGB camera-only tracking.
Fusion of Thermal Infrared and Visible Spectrum Video for Robust Surveillance
Computer Vision, Graphics and Image Processing, 2006
This paper presents an approach of fusing the information provided by visible spectrum video with that of thermal infrared video to tackle video processing challenges such as object detection and tracking for increasing the performance and robustness of the surveillance system. An enhanced object detection strategy using gradient information along with background subtraction is implemented with efficient fusion based approach to handle typical problems in both the domains. An intelligent fusion approach using Fuzzy logic and Kalman filtering technique is proposed to track objects and obtain fused estimate according to the reliability of the sensors. Appropriate measurement parameters are identified to determine the measurement accuracy of each sensor. Experimental results are shown on some typical scenarios of detection and tracking of pedestrians.
Fusion of color and infrared video for moving human detection
Pattern Recognition, 2007
We approach the task of human silhouette extraction from color and thermal image sequences using automatic image registration. Image registration between color and thermal images is a challenging problem due to the difficulties associated with finding correspondence. However, the moving people in a static scene provide cues to address this problem. In this paper, we propose a hierarchical scheme to automatically find the correspondence between the preliminary human silhouettes extracted from synchronous color and thermal image sequences for image registration. Next, we discuss strategies for probabilistically combining cues from registered color and thermal images for improved human silhouette detection. It is shown that the proposed approach achieves good results for image registration and human silhouette extraction. Experimental results also show a comparison of various sensor fusion strategies and demonstrate the improvement in performance over nonfused cases for human silhouette extraction.
, In Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXIII, edited by Gerald C. Holst, Keith A. Krapels, Proceedings of SPIE vol. 8355 (SPIE, Bellingham, WA, 2012) CID Number 83551B, Baltimore, USA
In law enforcement and security applications, the acquisition of face images is critical in producing key trace evidence for the successful identification of potential threats. In this work we, first, use a near infrared (NIR) sensor designed with the capability to acquire images at middle-range stand-off distances at night. Then, we determine the maximum stand-off distance where face recognition techniques can be utilized to efficiently recognize individuals at night at ranges from 30 to approximately 300 ft. The focus of the study is on establishing the maximum capabilities of the mid-range sensor to acquire good quality face images necessary for recognition. For the purpose of this study, a database in the visible (baseline) and NIR spectrum of 103 subjects is assembled and used to illustrate the challenges associated with the problem. In order to perform matching studies, we use multiple face recognition techniques and demonstrate that certain techniques are more robust in terms of recognition performance when using face images acquired at different distances. Experiments show that matching NIR face images at longer ranges (i.e. greater than about 300 feet or 90 meters using our camera system) is a very challenging problem and it requires further investigation.
Low Resolution Person Detection with a Moving Thermal Infrared Camera by Hot Spot Classification
In many visual surveillance applications the task of person detection and localization can be solved easier by using thermal long-wave infrared (LWIR) cameras which are less affected by changing illumination or background texture than visual-optical cameras. Especially in outdoor scenes where usually only few hot spots appear in thermal infrared imagery, humans can be detected more reliably due to their prominent infrared signature. We propose a two-stage person recognition approach for LWIR images: (1) the application of Maximally Stable Extremal Regions (MSER) to detect hot spots instead of background subtraction or sliding window and (2) the verification of the detected hot spots using a Discrete Cosine Transform (DCT) based descrip-tor and a modified Random Na¨ıve Bayes (RNB) classifier. The main contributions are the novel modified RNB clas-sifier and the generality of our method. We achieve high detection rates for several different LWIR datasets with low resolution videos in real-time. While many papers in this topic are dealing with strong constraints such as considering only one dataset, assuming a stationary camera, or detecting only moving persons, we aim at avoiding such constraints to make our approach applicable with moving platforms such as Unmanned Ground Vehicles (UGV).