Wenyan Jia - Academia.edu (original) (raw)

Papers by Wenyan Jia

IEEE Transactions on Cybernetics

Camera-based passive dietary intake monitoring is able to continuously capture the eating episode... more Camera-based passive dietary intake monitoring is able to continuously capture the eating episodes of a subject, recording rich visual information, such as the type and volume of food being consumed, as well as the eating behaviors of the subject. However, there currently is no method that is able to incorporate these visual clues and provide a comprehensive context of dietary intake from passive recording (e.g., is the subject sharing food with others, what food the subject is eating, and how much food is left in the bowl). On the other hand, privacy is a major concern while egocentric wearable cameras are used for capturing. In this article, we propose a privacy-preserved secure solution (i.e., egocentric image captioning) for dietary assessment with passive monitoring, which unifies food recognition, volume estimation, and scene understanding. By converting images into rich text descriptions, nutritionists can assess individual dietary intake based on the captions instead of the original images, reducing the risk of privacy leakage from images. To this end, an egocentric dietary image captioning dataset has been

Frontiers in Artificial Intelligence, 2021

Malnutrition, including both undernutrition and obesity, is a significant problem in low- and mid... more Malnutrition, including both undernutrition and obesity, is a significant problem in low- and middle-income countries (LMICs). In order to study malnutrition and develop effective intervention strategies, it is crucial to evaluate nutritional status in LMICs at the individual, household, and community levels. In a multinational research project supported by the Bill & Melinda Gates Foundation, we have been using a wearable technology to conduct objective dietary assessment in sub-Saharan Africa. Our assessment includes multiple diet-related activities in urban and rural families, including food sources (e.g., shopping, harvesting, and gathering), preservation/storage, preparation, cooking, and consumption (e.g., portion size and nutrition analysis). Our wearable device (“eButton” worn on the chest) acquires real-life images automatically during wake hours at preset time intervals. The recorded images, in amounts of tens of thousands per day, are post-processed to obtain the informat...

Background: It is well-known that many chronic diseases are associated with unhealthy diet. Altho... more Background: It is well-known that many chronic diseases are associated with unhealthy diet. Although improving diet is critical, adopting a healthy diet is difficult despite its benefits being well understood. Technology is needed that allows assessment of dietary intake accurately and easily in real-world settings so that effective intervention to manage overweight, obesity and related chronic diseases can be developed. In recent years, new wearable imaging and computational technologies have emerged. These technologies are capable of objective and passive dietary assessment with much simplified procedure than traditional questionnaires. However, a critical task is required to estimate the portion size (in this case, the food volume) from a digital image. Currently, this task is very challenging because the volumetric information in the two-dimensional images is incomplete, and the estimation involves a great deal of imagination, beyond the capacity of the traditional image process...

EURASIP Journal on Advances in Signal Processing, 2019

Recently, egocentric activity recognition has attracted considerable attention in the pattern rec... more Recently, egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its widespread applicability to human systems, including the evaluation of dietary and physical activity and the monitoring of patients and older adults. In this paper, we present a knowledge-driven multisource fusion framework for the recognition of egocentric activities in daily living (ADL). This framework employs Dezert-Smarandache theory across three information sources: the wearer's knowledge, images acquired by a wearable camera, and sensor data from wearable inertial measurement units and GPS. A simple likelihood table is designed to provide routine ADL information for each individual. A well-trained convolutional neural network is then used to produce a set of textual tags that, along with routine information and other sensor data, are used to recognize ADLs based on information theory-based statistics and a support vector machine. Our experiments show that the proposed method accurately recognizes 15 predefined ADL classes, including a variety of sedentary activities that have previously been difficult to recognize. When applied to real-life data recorded using a self-constructed wearable device, our method outperforms previous approaches, and an average accuracy of 85.4% is achieved for the 15 ADLs.

Nutrition Journal, 2018

Background: Food preparation skills may encourage healthy eating. Traditional assessment of child... more Background: Food preparation skills may encourage healthy eating. Traditional assessment of child food preparation employs self-or parent proxy-reporting methods, which are prone to error. The eButton is a wearable all-day camera that has promise as an objective, passive method for measuring child food preparation practices. Purpose: This paper explores the feasibility of the eButton to reliably capture home food preparation behaviors and practices in a sample of pre-and early adolescents (ages 9 to 13). Methods: This is a secondary analysis of two eButton pilot projects evaluating the dietary intake of pre-and early adolescents in or around Houston, Texas. Food preparation behaviors were coded into seven major categories including: browsing, altering food/adding seasoning, food media, meal related tasks, prep work, cooking and observing. Inter-coder reliability was measured using Cohen's kappa and percent agreement. Results: Analysis was completed on data for 31 participants. The most common activity was browsing in the pantry or fridge. Few participants demonstrated any food preparation work beyond unwrapping of food packages and combining two or more ingredients; actual cutting or measuring of foods were rare. Conclusions: Although previous research suggests children who "help" prepare meals may obtain some dietary benefit, accurate assessment tools of food preparation behavior are lacking. The eButton offers a feasible approach to food preparation behavior measurement among pre-and early adolescents. Follow up research exploring the validity of this method in a larger sample, and comparisons between cooking behavior and dietary intake are needed.

Public health nutrition, Jan 6, 2018

Current approaches to food volume estimation require the person to carry a fiducial marker (e.g. ... more Current approaches to food volume estimation require the person to carry a fiducial marker (e.g. a checkerboard card), to be placed next to the food before taking a picture. This procedure is inconvenient and post-processing of the food picture is time-consuming and sometimes inaccurate. These problems keep people from using the smartphone for self-administered dietary assessment. The current bioengineering study presents a novel smartphone-based imaging approach to table-side estimation of food volume which overcomes current limitations. We present a new method for food volume estimation without a fiducial marker. Our mathematical model indicates that, using a special picture-taking strategy, the smartphone-based imaging system can be calibrated adequately if the physical length of the smartphone and the output of the motion sensor within the device are known. We also present and test a new virtual reality method for food volume estimation using the International Food Unit™ and a t...

Public health nutrition, Jan 12, 2018

The eButton takes frontal images at 4s intervals throughout the day. A three-dimensional manually... more The eButton takes frontal images at 4s intervals throughout the day. A three-dimensional manually administered wire mesh procedure has been developed to quantify portion sizes from the two-dimensional images. The present paper reports a test of the inter-rater reliability and validity of use of the wire mesh procedure. Seventeen foods of diverse shapes and sizes served on plates, bowls and cups were selected to rigorously test the portion assessment procedure. A dietitian not involved in inter-rater reliability assessment used standard cups to independently measure the quantities of foods to generate the 'true' value for a total of seventy-five 'served' and seventy-five smaller 'left' images with diverse portion sizes. The images appeared on the computer to which the digital wire meshes were applied. Two dietitians and three engineers independently estimated portion size of the larger ('served') and smaller ('left') images for the same foods. ...

2015 41st Annual Northeast Biomedical Engineering Conference (NEBEC), 2015

This paper presents an image-based indoor localization system for tracking older individuals' mov... more This paper presents an image-based indoor localization system for tracking older individuals' movement at home. In this system, images are acquired at a low frame rate by a miniature camera worn conveniently at the chest position. The correspondence between adjacent frames is first established by matching the SIFT (scale-invariant feature transform) based key points in a pair of images. The location changes of these points are then used to estimate the position of the wearer based on use of the pinhole camera model. A preliminary study conducted in an indoor environment indicates that the location of the wearer can be estimated with an adequate accuracy.

Proceedings of the IEEE ... annual Northeast Bioengineering Conference. IEEE Northeast Bioengineering Conference, 2015

In this paper, an efficient field-programmable gate array (FPGA) implementation of the JPEG basel... more In this paper, an efficient field-programmable gate array (FPGA) implementation of the JPEG baseline image compression encoder is presented for wearable devices in health and wellness applications. In order to gain flexibility in developing FPGA-specific software and balance between real-time performance and resources utilization, A High Level Synthesis (HLS) tool is utilized in our system design. An optimized dataflow configuration with a padding scheme simplifies the timing control for data transfer. Our experiments with a system-on-chip multi-sensor system have verified our FPGA implementation with respect to real-time performance, computational efficiency, and FPGA resource utilization.

Optik - International Journal for Light and Electron Optics, 2015

We propose a novel object tracking framework based on online learning scheme that can work robust... more We propose a novel object tracking framework based on online learning scheme that can work robustly in challenging scenarios. Firstly, a learning-based particle filter is proposed with color and edge-based features. We train a. support vector machine (SVM) classifier with object and background information and map the outputs into probabilities, then the weight of particles in a particle filter can be calculated by the probabilistic outputs to estimate the state of the object. Secondly, the tracking loop starts with Lucas-Kanade (LK) affine template matching and follows by learning-based particle filter tracking. Lucas-Kanade method estimates errors and updates object template in the positive samples dataset, and learning-based particle filter tracker will start if the LK tracker loses the object. Finally, SVM classifier evaluates every tracked appearance to update the training set or restart the tracking loop if necessary. Experimental results show that our method is robust to challenging light, scale and pose changing, and test on eButton image sequence also achieves satisfactory tracking performance.

2015 41st Annual Northeast Biomedical Engineering Conference (NEBEC), 2015

This study investigates the use of a chest-worn wearable computer, the eButton, to assess physica... more This study investigates the use of a chest-worn wearable computer, the eButton, to assess physical performance of older adults. The Short Physical Performance Battery (SPPB), a standard cliniucal test, is first conducted on older human subjects. Then, a triaxial accelerometer and a triaxial gyroscope within the eButton are utilized to record acceleration and angular velocity of body motion on the same subjects for one week. The sensor data corresponding to walking episodes are segmented and features in the time and frequency domains are extracted. Comparison between these features and the total SPPB scores shows that the sensor data acquired in free-living conditions can be used as indicators of the subjects physical performance.

Proceedings of the 4th International SenseCam & Pervasive Imaging Conference on - SenseCam '13, 2013

We present an eating activity detection method via automatic detecting dining plates from images ... more We present an eating activity detection method via automatic detecting dining plates from images acquired chronically by a wearable camera. Convex edge segments and their combinations within each input image are modeled with respect to probabilities of belonging to candidate ellipses. Then, a dining plate is determined according to a confidence score. Finally, the presence/absence of an eating event in an image sequence is determined by analyzing successive frames. Our experimental results verified the effectiveness of this method.

Measurement Science and Technology, 2015

Image-based dietary assessment has recently received much attention in the community of obesity r... more Image-based dietary assessment has recently received much attention in the community of obesity research. In this assessment, foods in digital pictures are specified, and their portion sizes (volumes) are estimated. Although manual processing is currently the most utilized method, image processing holds much promise since it may eventually lead to automatic dietary assessment. In this paper we study the problem of segmenting food objects from images. This segmentation is difficult because of various food types, shapes and colors, different decorating patterns on food containers, and occlusions of food and non-food objects. We propose a novel method based on a saliency-aware active contour model (ACM) for automatic food segmentation from images acquired by a wearable camera. An integrated saliency estimation approach based on food location priors and visual attention features is designed to produce a salient map of possible food regions in the input image. Next, a geometric contour primitive is generated and fitted to the salient map by means of multi-resolution optimization with respect to a set of affine and elastic transformation parameters. The food regions are then extracted after contour fitting. Our experiments using 60 food images showed that the proposed method achieved significantly higher accuracy in food segmentation when compared to conventional segmentation methods.

2011 IEEE 37th Annual Northeast Bioengineering Conference (NEBEC), 2011

Obesity has become a widespread epidemic threatening the health of millions of Americans and cost... more Obesity has become a widespread epidemic threatening the health of millions of Americans and costing billions of dollars in health care. In both obesity research and clinical intervention, an accurate tool for diet evaluation is required. In this thesis, a new approach to the estimation of the volume of food from a single input image is presented based on the virtual reality (VR) technology. A software system is constructed for food image acquisition, camera parameters calibration, virtual reality modeling, virtual object manipulation, and food volume estimation. Our system utilizes a checkerboard to calibrate the intrinsic and extrinsic parameters of the camera using image process techniques. Once these parameters are obtained, we establish a VR space in which a virtual 3D wireframe is projected into the food image in a well-defined proportional relationship. Within this space, the user is able to scale, deform, translate and rotate the virtual wireframe to fit the food in the image. Finally, the known volume of the wireframe is utilized to compute the food volume using the proportional relationship. Our experimental study has indicated that our VR system is highly accurate and robust in estimating volumes of both regularly and irregularly shaped foods, providing a powerful diet evaluation tool for both obesity research and treatment.

Proceedings of the IEEE ... annual Northeast Bioengineering Conference. IEEE Northeast Bioengineering Conference, Jan 31, 2012

Image-based dietary assessment is important for health monitoring and management because it can p... more Image-based dietary assessment is important for health monitoring and management because it can provide quantitative and objective information, such as food volume, nutrition type, and calorie intake. In this paper, a new framework, 3D/2D model-to-image registration, is presented for estimating food volume from a single-view 2D image containing a reference object (i.e., a circular dining plate). First, the food is segmented from the background image based on Otsu's thresholding and morphological operations. Next, the food volume is obtained from a user-selected, 3D shape model. The position, orientation and scale of the model are optimized by a model-to-image registration process. Then, the circular plate in the image is fitted and its spatial information is used as constraints for solving the registration problem. Our method takes the global contour information of the shape model into account to obtain a reliable food volume estimate. Experimental results using regularly shaped...

Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference, 2012

Food portion size measurement combined with a database of calories and nutrients is important in ... more Food portion size measurement combined with a database of calories and nutrients is important in the study of metabolic disorders such as obesity and diabetes. In this work, we present a convenient and accurate approach to the calculation of food volume by measuring several dimensions using a single 2-D image as the input. This approach does not require the conventional checkerboard based camera calibration since it is burdensome in practice. The only prior requirements of our approach are: 1) a circular container with a known size, such as a plate, a bowl or a cup, is present in the image, and 2) the picture is taken under a reasonable assumption that the camera is always held level with respect to its left and right sides and its lens is tilted down towards foods on the dining table. We show that, under these conditions, our approach provides a closed form solution to camera calibration, allowing convenient measurement of food portion size using digital pictures.

2011 4th International Congress on Image and Signal Processing, 2011

We present a new method to recognize activity patterns from video acquired by a camera mounted on... more We present a new method to recognize activity patterns from video acquired by a camera mounted on the target (i.e., activity performer). Because of this unconventional camera setting, algorithms for activity recognition must be redesigned because the activity performer never appears in the video. We approach this recognition problem indirectly by observing background changes in the acquired image sequences. A motion histogram scheme is proposed to characterize activity patterns from the perspective of camera motion. This histogram is utilized as the input to our activity identification algorithm based on a hidden Markov model. Our experimental results show that our method successfully identifies complex activities even the motion profile of an activity involves a large variance. Our method is applied to the construction of a new wearable device that helps people lose weight and maintain a healthy lifestyle by automatically recognizing and monitoring physical activity.

2012 38th Annual Northeast Bioengineering Conference (NEBEC), 2012

A wearable computer, called eButton, has been developed for evaluation of the human lifestyle. Th... more A wearable computer, called eButton, has been developed for evaluation of the human lifestyle. This ARM-based device acquires multimodal data from a camera module, a motion sensor, an orientation sensor, a light sensor and a GPS receiver. Its performance has been tested both in our laboratory and by human subjects in free-living conditions. Our results indicate that eButton can record real-world data reliably, providing a powerful tool for the evaluation of lifestyle for a broad range of applications.

2013 39th Annual Northeast Bioengineering Conference, 2013

Eating event detection is an important problem in automatic dietary study using a wearable comput... more Eating event detection is an important problem in automatic dietary study using a wearable computer, such as the eButton. In this work, we approach this detection problem based on the use of a small magnet marker attached to a finger and a miniature magnetometer installed within the eButton. Our experimental results indicate that our magnetic approach is effective when the distance between the marker and the wearable computer is within 12cm, and the range of detection is approximately 15cm. We also found that the proximity signal patterns corresponding to eating and other daily activities are different, which can be used to reduce the false detection rate. In addition, our approach is convenient, low-cost and energy efficient, suitable for practical applications.

Proceedings of the 2010 IEEE 36th Annual Northeast Bioengineering Conference (NEBEC), 2010

This paper presents an automatic video analysis method for physical activity classification and m... more This paper presents an automatic video analysis method for physical activity classification and measurement. A wearable device is used to capture daily life data for health monitoring. Physical activity is analyzed by using the change of surrounding scenes resulting from the motion of the wearer. Recognition of different physical activities is achieved by analyzing motion characteristics in images evaluated from a set of representative pixel pairs extracted from adjacent video frames. Ambiguous and incorrect pixel pairs are removed under the epipolar constraint from stereo images. The effectiveness of the new method is demonstrated through experiments. I.

IEEE Transactions on Cybernetics

Camera-based passive dietary intake monitoring is able to continuously capture the eating episode... more Camera-based passive dietary intake monitoring is able to continuously capture the eating episodes of a subject, recording rich visual information, such as the type and volume of food being consumed, as well as the eating behaviors of the subject. However, there currently is no method that is able to incorporate these visual clues and provide a comprehensive context of dietary intake from passive recording (e.g., is the subject sharing food with others, what food the subject is eating, and how much food is left in the bowl). On the other hand, privacy is a major concern while egocentric wearable cameras are used for capturing. In this article, we propose a privacy-preserved secure solution (i.e., egocentric image captioning) for dietary assessment with passive monitoring, which unifies food recognition, volume estimation, and scene understanding. By converting images into rich text descriptions, nutritionists can assess individual dietary intake based on the captions instead of the original images, reducing the risk of privacy leakage from images. To this end, an egocentric dietary image captioning dataset has been

Frontiers in Artificial Intelligence, 2021

Malnutrition, including both undernutrition and obesity, is a significant problem in low- and mid... more Malnutrition, including both undernutrition and obesity, is a significant problem in low- and middle-income countries (LMICs). In order to study malnutrition and develop effective intervention strategies, it is crucial to evaluate nutritional status in LMICs at the individual, household, and community levels. In a multinational research project supported by the Bill & Melinda Gates Foundation, we have been using a wearable technology to conduct objective dietary assessment in sub-Saharan Africa. Our assessment includes multiple diet-related activities in urban and rural families, including food sources (e.g., shopping, harvesting, and gathering), preservation/storage, preparation, cooking, and consumption (e.g., portion size and nutrition analysis). Our wearable device (“eButton” worn on the chest) acquires real-life images automatically during wake hours at preset time intervals. The recorded images, in amounts of tens of thousands per day, are post-processed to obtain the informat...

Background: It is well-known that many chronic diseases are associated with unhealthy diet. Altho... more Background: It is well-known that many chronic diseases are associated with unhealthy diet. Although improving diet is critical, adopting a healthy diet is difficult despite its benefits being well understood. Technology is needed that allows assessment of dietary intake accurately and easily in real-world settings so that effective intervention to manage overweight, obesity and related chronic diseases can be developed. In recent years, new wearable imaging and computational technologies have emerged. These technologies are capable of objective and passive dietary assessment with much simplified procedure than traditional questionnaires. However, a critical task is required to estimate the portion size (in this case, the food volume) from a digital image. Currently, this task is very challenging because the volumetric information in the two-dimensional images is incomplete, and the estimation involves a great deal of imagination, beyond the capacity of the traditional image process...

EURASIP Journal on Advances in Signal Processing, 2019

Recently, egocentric activity recognition has attracted considerable attention in the pattern rec... more Recently, egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its widespread applicability to human systems, including the evaluation of dietary and physical activity and the monitoring of patients and older adults. In this paper, we present a knowledge-driven multisource fusion framework for the recognition of egocentric activities in daily living (ADL). This framework employs Dezert-Smarandache theory across three information sources: the wearer's knowledge, images acquired by a wearable camera, and sensor data from wearable inertial measurement units and GPS. A simple likelihood table is designed to provide routine ADL information for each individual. A well-trained convolutional neural network is then used to produce a set of textual tags that, along with routine information and other sensor data, are used to recognize ADLs based on information theory-based statistics and a support vector machine. Our experiments show that the proposed method accurately recognizes 15 predefined ADL classes, including a variety of sedentary activities that have previously been difficult to recognize. When applied to real-life data recorded using a self-constructed wearable device, our method outperforms previous approaches, and an average accuracy of 85.4% is achieved for the 15 ADLs.

Nutrition Journal, 2018

Background: Food preparation skills may encourage healthy eating. Traditional assessment of child... more Background: Food preparation skills may encourage healthy eating. Traditional assessment of child food preparation employs self-or parent proxy-reporting methods, which are prone to error. The eButton is a wearable all-day camera that has promise as an objective, passive method for measuring child food preparation practices. Purpose: This paper explores the feasibility of the eButton to reliably capture home food preparation behaviors and practices in a sample of pre-and early adolescents (ages 9 to 13). Methods: This is a secondary analysis of two eButton pilot projects evaluating the dietary intake of pre-and early adolescents in or around Houston, Texas. Food preparation behaviors were coded into seven major categories including: browsing, altering food/adding seasoning, food media, meal related tasks, prep work, cooking and observing. Inter-coder reliability was measured using Cohen's kappa and percent agreement. Results: Analysis was completed on data for 31 participants. The most common activity was browsing in the pantry or fridge. Few participants demonstrated any food preparation work beyond unwrapping of food packages and combining two or more ingredients; actual cutting or measuring of foods were rare. Conclusions: Although previous research suggests children who "help" prepare meals may obtain some dietary benefit, accurate assessment tools of food preparation behavior are lacking. The eButton offers a feasible approach to food preparation behavior measurement among pre-and early adolescents. Follow up research exploring the validity of this method in a larger sample, and comparisons between cooking behavior and dietary intake are needed.

Public health nutrition, Jan 6, 2018

Current approaches to food volume estimation require the person to carry a fiducial marker (e.g. ... more Current approaches to food volume estimation require the person to carry a fiducial marker (e.g. a checkerboard card), to be placed next to the food before taking a picture. This procedure is inconvenient and post-processing of the food picture is time-consuming and sometimes inaccurate. These problems keep people from using the smartphone for self-administered dietary assessment. The current bioengineering study presents a novel smartphone-based imaging approach to table-side estimation of food volume which overcomes current limitations. We present a new method for food volume estimation without a fiducial marker. Our mathematical model indicates that, using a special picture-taking strategy, the smartphone-based imaging system can be calibrated adequately if the physical length of the smartphone and the output of the motion sensor within the device are known. We also present and test a new virtual reality method for food volume estimation using the International Food Unit™ and a t...

Public health nutrition, Jan 12, 2018

The eButton takes frontal images at 4s intervals throughout the day. A three-dimensional manually... more The eButton takes frontal images at 4s intervals throughout the day. A three-dimensional manually administered wire mesh procedure has been developed to quantify portion sizes from the two-dimensional images. The present paper reports a test of the inter-rater reliability and validity of use of the wire mesh procedure. Seventeen foods of diverse shapes and sizes served on plates, bowls and cups were selected to rigorously test the portion assessment procedure. A dietitian not involved in inter-rater reliability assessment used standard cups to independently measure the quantities of foods to generate the 'true' value for a total of seventy-five 'served' and seventy-five smaller 'left' images with diverse portion sizes. The images appeared on the computer to which the digital wire meshes were applied. Two dietitians and three engineers independently estimated portion size of the larger ('served') and smaller ('left') images for the same foods. ...

2015 41st Annual Northeast Biomedical Engineering Conference (NEBEC), 2015

This paper presents an image-based indoor localization system for tracking older individuals' mov... more This paper presents an image-based indoor localization system for tracking older individuals' movement at home. In this system, images are acquired at a low frame rate by a miniature camera worn conveniently at the chest position. The correspondence between adjacent frames is first established by matching the SIFT (scale-invariant feature transform) based key points in a pair of images. The location changes of these points are then used to estimate the position of the wearer based on use of the pinhole camera model. A preliminary study conducted in an indoor environment indicates that the location of the wearer can be estimated with an adequate accuracy.

Proceedings of the IEEE ... annual Northeast Bioengineering Conference. IEEE Northeast Bioengineering Conference, 2015

In this paper, an efficient field-programmable gate array (FPGA) implementation of the JPEG basel... more In this paper, an efficient field-programmable gate array (FPGA) implementation of the JPEG baseline image compression encoder is presented for wearable devices in health and wellness applications. In order to gain flexibility in developing FPGA-specific software and balance between real-time performance and resources utilization, A High Level Synthesis (HLS) tool is utilized in our system design. An optimized dataflow configuration with a padding scheme simplifies the timing control for data transfer. Our experiments with a system-on-chip multi-sensor system have verified our FPGA implementation with respect to real-time performance, computational efficiency, and FPGA resource utilization.

Optik - International Journal for Light and Electron Optics, 2015

We propose a novel object tracking framework based on online learning scheme that can work robust... more We propose a novel object tracking framework based on online learning scheme that can work robustly in challenging scenarios. Firstly, a learning-based particle filter is proposed with color and edge-based features. We train a. support vector machine (SVM) classifier with object and background information and map the outputs into probabilities, then the weight of particles in a particle filter can be calculated by the probabilistic outputs to estimate the state of the object. Secondly, the tracking loop starts with Lucas-Kanade (LK) affine template matching and follows by learning-based particle filter tracking. Lucas-Kanade method estimates errors and updates object template in the positive samples dataset, and learning-based particle filter tracker will start if the LK tracker loses the object. Finally, SVM classifier evaluates every tracked appearance to update the training set or restart the tracking loop if necessary. Experimental results show that our method is robust to challenging light, scale and pose changing, and test on eButton image sequence also achieves satisfactory tracking performance.

2015 41st Annual Northeast Biomedical Engineering Conference (NEBEC), 2015

This study investigates the use of a chest-worn wearable computer, the eButton, to assess physica... more This study investigates the use of a chest-worn wearable computer, the eButton, to assess physical performance of older adults. The Short Physical Performance Battery (SPPB), a standard cliniucal test, is first conducted on older human subjects. Then, a triaxial accelerometer and a triaxial gyroscope within the eButton are utilized to record acceleration and angular velocity of body motion on the same subjects for one week. The sensor data corresponding to walking episodes are segmented and features in the time and frequency domains are extracted. Comparison between these features and the total SPPB scores shows that the sensor data acquired in free-living conditions can be used as indicators of the subjects physical performance.

Proceedings of the 4th International SenseCam & Pervasive Imaging Conference on - SenseCam '13, 2013

We present an eating activity detection method via automatic detecting dining plates from images ... more We present an eating activity detection method via automatic detecting dining plates from images acquired chronically by a wearable camera. Convex edge segments and their combinations within each input image are modeled with respect to probabilities of belonging to candidate ellipses. Then, a dining plate is determined according to a confidence score. Finally, the presence/absence of an eating event in an image sequence is determined by analyzing successive frames. Our experimental results verified the effectiveness of this method.

Measurement Science and Technology, 2015

Image-based dietary assessment has recently received much attention in the community of obesity r... more Image-based dietary assessment has recently received much attention in the community of obesity research. In this assessment, foods in digital pictures are specified, and their portion sizes (volumes) are estimated. Although manual processing is currently the most utilized method, image processing holds much promise since it may eventually lead to automatic dietary assessment. In this paper we study the problem of segmenting food objects from images. This segmentation is difficult because of various food types, shapes and colors, different decorating patterns on food containers, and occlusions of food and non-food objects. We propose a novel method based on a saliency-aware active contour model (ACM) for automatic food segmentation from images acquired by a wearable camera. An integrated saliency estimation approach based on food location priors and visual attention features is designed to produce a salient map of possible food regions in the input image. Next, a geometric contour primitive is generated and fitted to the salient map by means of multi-resolution optimization with respect to a set of affine and elastic transformation parameters. The food regions are then extracted after contour fitting. Our experiments using 60 food images showed that the proposed method achieved significantly higher accuracy in food segmentation when compared to conventional segmentation methods.

2011 IEEE 37th Annual Northeast Bioengineering Conference (NEBEC), 2011

Obesity has become a widespread epidemic threatening the health of millions of Americans and cost... more Obesity has become a widespread epidemic threatening the health of millions of Americans and costing billions of dollars in health care. In both obesity research and clinical intervention, an accurate tool for diet evaluation is required. In this thesis, a new approach to the estimation of the volume of food from a single input image is presented based on the virtual reality (VR) technology. A software system is constructed for food image acquisition, camera parameters calibration, virtual reality modeling, virtual object manipulation, and food volume estimation. Our system utilizes a checkerboard to calibrate the intrinsic and extrinsic parameters of the camera using image process techniques. Once these parameters are obtained, we establish a VR space in which a virtual 3D wireframe is projected into the food image in a well-defined proportional relationship. Within this space, the user is able to scale, deform, translate and rotate the virtual wireframe to fit the food in the image. Finally, the known volume of the wireframe is utilized to compute the food volume using the proportional relationship. Our experimental study has indicated that our VR system is highly accurate and robust in estimating volumes of both regularly and irregularly shaped foods, providing a powerful diet evaluation tool for both obesity research and treatment.

Proceedings of the IEEE ... annual Northeast Bioengineering Conference. IEEE Northeast Bioengineering Conference, Jan 31, 2012

Image-based dietary assessment is important for health monitoring and management because it can p... more Image-based dietary assessment is important for health monitoring and management because it can provide quantitative and objective information, such as food volume, nutrition type, and calorie intake. In this paper, a new framework, 3D/2D model-to-image registration, is presented for estimating food volume from a single-view 2D image containing a reference object (i.e., a circular dining plate). First, the food is segmented from the background image based on Otsu's thresholding and morphological operations. Next, the food volume is obtained from a user-selected, 3D shape model. The position, orientation and scale of the model are optimized by a model-to-image registration process. Then, the circular plate in the image is fitted and its spatial information is used as constraints for solving the registration problem. Our method takes the global contour information of the shape model into account to obtain a reliable food volume estimate. Experimental results using regularly shaped...

Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference, 2012

Food portion size measurement combined with a database of calories and nutrients is important in ... more Food portion size measurement combined with a database of calories and nutrients is important in the study of metabolic disorders such as obesity and diabetes. In this work, we present a convenient and accurate approach to the calculation of food volume by measuring several dimensions using a single 2-D image as the input. This approach does not require the conventional checkerboard based camera calibration since it is burdensome in practice. The only prior requirements of our approach are: 1) a circular container with a known size, such as a plate, a bowl or a cup, is present in the image, and 2) the picture is taken under a reasonable assumption that the camera is always held level with respect to its left and right sides and its lens is tilted down towards foods on the dining table. We show that, under these conditions, our approach provides a closed form solution to camera calibration, allowing convenient measurement of food portion size using digital pictures.

2011 4th International Congress on Image and Signal Processing, 2011

We present a new method to recognize activity patterns from video acquired by a camera mounted on... more We present a new method to recognize activity patterns from video acquired by a camera mounted on the target (i.e., activity performer). Because of this unconventional camera setting, algorithms for activity recognition must be redesigned because the activity performer never appears in the video. We approach this recognition problem indirectly by observing background changes in the acquired image sequences. A motion histogram scheme is proposed to characterize activity patterns from the perspective of camera motion. This histogram is utilized as the input to our activity identification algorithm based on a hidden Markov model. Our experimental results show that our method successfully identifies complex activities even the motion profile of an activity involves a large variance. Our method is applied to the construction of a new wearable device that helps people lose weight and maintain a healthy lifestyle by automatically recognizing and monitoring physical activity.

2012 38th Annual Northeast Bioengineering Conference (NEBEC), 2012

A wearable computer, called eButton, has been developed for evaluation of the human lifestyle. Th... more A wearable computer, called eButton, has been developed for evaluation of the human lifestyle. This ARM-based device acquires multimodal data from a camera module, a motion sensor, an orientation sensor, a light sensor and a GPS receiver. Its performance has been tested both in our laboratory and by human subjects in free-living conditions. Our results indicate that eButton can record real-world data reliably, providing a powerful tool for the evaluation of lifestyle for a broad range of applications.

2013 39th Annual Northeast Bioengineering Conference, 2013

Eating event detection is an important problem in automatic dietary study using a wearable comput... more Eating event detection is an important problem in automatic dietary study using a wearable computer, such as the eButton. In this work, we approach this detection problem based on the use of a small magnet marker attached to a finger and a miniature magnetometer installed within the eButton. Our experimental results indicate that our magnetic approach is effective when the distance between the marker and the wearable computer is within 12cm, and the range of detection is approximately 15cm. We also found that the proximity signal patterns corresponding to eating and other daily activities are different, which can be used to reduce the false detection rate. In addition, our approach is convenient, low-cost and energy efficient, suitable for practical applications.

Proceedings of the 2010 IEEE 36th Annual Northeast Bioengineering Conference (NEBEC), 2010

This paper presents an automatic video analysis method for physical activity classification and m... more This paper presents an automatic video analysis method for physical activity classification and measurement. A wearable device is used to capture daily life data for health monitoring. Physical activity is analyzed by using the change of surrounding scenes resulting from the motion of the wearer. Recognition of different physical activities is achieved by analyzing motion characteristics in images evaluated from a set of representative pixel pairs extracted from adjacent video frames. Ambiguous and incorrect pixel pairs are removed under the epipolar constraint from stereo images. The effectiveness of the new method is demonstrated through experiments. I.