Ensembles of Deep LSTM Learners for Activity Recognition using Wearables (original) (raw)
Article No.: 11, Pages 1 - 28
Published: 30 June 2017 Publication History
Abstract
Recently, deep learning (DL) methods have been introduced very successfully into human activity recognition (HAR) scenarios in ubiquitous and wearable computing. Especially the prospect of overcoming the need for manual feature design combined with superior classification capabilities render deep neural networks very attractive for real-life HAR applications. Even though DL-based approaches now outperform the state-of-the-art in a number of recognition tasks, still substantial challenges remain. Most prominently, issues with real-life datasets, typically including imbalanced datasets and problematic data quality, still limit the effectiveness of activity recognition using wearables. In this paper we tackle such challenges through Ensembles of deep Long Short Term Memory (LSTM) networks. LSTM networks currently represent the state-of-the-art with superior classification performance on relevant HAR benchmark datasets. We have developed modified training procedures for LSTM networks and combine sets of diverse LSTM learners into classifier collectives. We demonstrate that Ensembles of deep LSTM learners outperform individual LSTM networks and thus push the state-of-the-art in human activity recognition using wearables. Through an extensive experimental evaluation on three standard benchmarks (Opportunity, PAMAP2, Skoda) we demonstrate the excellent recognition capabilities of our approach and its potential for real-life applications of human activity recognition.
References
[1]
Gregory D Abowd. 2016. Beyond Weiser - From Ubiquitous to Collective Computing. IEEE Computer 49, 1 (2016), 17--23.
[2]
Mohammad A Alsheikh, Ahmed Selim, Dusit Niyato, Linda Doyle, Shaowei Lin, and Hwee-Pink Tan. 2016. Deep Activity Recognition Models with Triaxial Accelerometers. In Proc. AAAI Workshop on Artificial Intelligence Applied to Assistive Technologies and Smart Environments.
[3]
Akin Avci, Stephan Bosch, Mihai Marin-Perianu, Raluca Marin-Perianu, and Paul Havinga. 2010. Activity Recognition Using Inertial Sensing for Healthcare, Wellbeing and Sports Applications: A Survey. In Proc. Int. Conf. Architecture of Compu. Systems (ARCS).
[4]
Marc Bachlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jeffrey M Hausdorff, Nir Giladi, and Gerhard Troster. 2010. Wearable assistant for Parkinson's disease patients with the freezing of gait symptom. IEEE Transactions on Information Technology in Biomedicine 14, 2 (2010), 436--446.
[5]
Sourav Bhattacharya and Nicholas D Lane. 2016. Sparsification and Separation of Deep Learning Layers for Constrained Resource Inference on Wearables. In Proc. Int. Conf. Embedded Networked Sensor Systems (SenSys).
[6]
Léon Bottou. 2010. Large-Scale Machine Learning with Stochastic Gradient Descent. In Prco. Int. Conf. Computational Statistics (COMPSTAT).
[7]
Leo Breiman. 1996. Bagging predictors. Machine Learning 24 (1996), 123--140.
[8]
Leo Breiman. 2001. Random Forests. Mach. Learn. (2001), 5--32.
[9]
Peter Bühlmann. 2012. Bagging, Boosting and Ensemble Methods. Springer Berlin Heidelberg, 985--1022.
[10]
Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. Comput. Surveys 46, 3 (Jan. 2014), 1--33.
[11]
Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José Del R Millán, and Daniel Roggen. 2013. The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (Nov. 2013), 2033--2042.
[12]
Linmi Chen, Jesse Hoey, Chris D Nugent, Diane J Cook, and Zhiwen Yu. 2012. Sensor-Based Activity Recognition. IEEE Trans. on Systems, Man, and Cybernetics - Part C: Applications and Reviews 42, 6 (2012), 790--808.
[13]
Krzysztof Dembczynski, Arkadiusz Jachnik, Wojciech Kotlowski, Willem Waegeman, and Eyke Hüllermeier. 2013. Optimizing the F-Measure in Multi-Label Classification: Plug-in Rule Approach versus Structured Loss Minimization. Li Deng and John C Platt. 2014. Ensemble deep learning for speech recognition. In Proc. INTERSPEECH.
[14]
Sander Dieleman and others. 2015. Lasagne: First release. (Aug. 2015).
[15]
Nils Y. Hammerla, James Fisher, Peter Andras, Lynn Rochester, Richard Walker, and Thomas Plötz. 2015. PD Disease State Assessment in Naturalistic Environments using Deep Learning. In Proc. Int. Conf. Assoc. Advancement of Art. Intelligence (AAAI).
[16]
Nils Y. Hammerla, Shane Halloran, and Thomas Plötz. 2016. Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables.
[17]
Nils Y. Hammerla, Reuben Kirkham, Peter Andras, and Thomas Plötz. 2013. On Preserving Statistical Characteristics of Accelerometry Data using their Empirical Cumulative Distribution.
[18]
Nils Y. Hammerla and Thomas Plötz. 2015. Let's (not) Stick Together: Pairwise Similarity Biases Cross-Validation in Activity Recognition.
[19]
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning. Springer.
[20]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation 9, 8 (Nov. 1997), 1735--1780.
[21]
Jesse Hoey, Thomas Plötz, Dan Jackson, Andrew Monk, Cuong Pham, and Patrick Olivier. 2011. Rapid specification and automated generation of prompting systems to assist people with dementia. Pervasive and Mobile Computing (PMC) 7, 3 (2011), 299--318.
[22]
Martin Jansche. 2005. Maximum Expected F-measure Training of Logistic Regression Models. In Proc. Int. Conf. on Human Language Technology and Empirical Methods in Natural Language Processing.
[23]
Aftab Khan, Sebastian Mellor, Eugen Berlin, Robin Thompson, Roisin McNaney, Patrick Olivier, and Thomas Plötz. 2015. Beyond Activity Recognition: Skill Assessment from Accelerometer Data.
[24]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proc. ACM Symp. User Interface and Software Technology (UIST).
[25]
Hyun-Chul Kim, Shaoning Pang, Hong-Mo Je, Daijin Kim, and Sung Yang Bang. 2003. Constructing support vector machine ensemble. Pattern Recognition 36, 12 (Dec. 2003), 2757--2767.
[26]
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learning Representation (IGLR).
[27]
Josef Kittler, Mohamad Hatef, Robert P W Duin, and Jiri Matas. 1998. On Combining Classifiers. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI) 20, 3 (1998), 226--239.
[28]
Matthias Kranz, Andreas Moeller, Nils Y. Hammerla, Stefan Diewald, Luis Roalter, Thomas Plötz, and Patrick Olivier. 2012. The mobile fitness coach: Towards individualized skill assessment using personalized mobile devices. Pervasive and Mobile Computing (PMC) (2012).
[29]
Ludmila Kuncheva. 2004. Combining pattern classifiers: methods and algorithms. John Wiley 8 Sons.
[30]
Cassim Ladha, Nils Y. Hammerla, Patrick Olivier, and Thomas Plötz. 2013. ClimbAX: Skill Assessment for Climbing Enthusiasts.
[31]
Nicholas D Lane, Petko Georgiev, and Lorena Qendro. 2015. DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning.
[32]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (May 2015), 436--444.
[33]
James L McClelland and David E Rumelhart. 1986. Parallel distributed processing. MIT Press.
[34]
Francisco Javier Ordóñez Morales and Daniel Roggen. 2016. Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations.
[35]
Francisco Ordóñez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors 16, 1 (Jan. 2016), 115.
[36]
Thomas Plötz, Nils Y. Hammerla, and Patrick Olivier. 2011. Feature Learning for Activity Recognition in Ubiquitous Computing.
[37]
Thomas Plötz, Nils Y. Hammerla, Agata Rozga, Andrea Reavis, Nathan Call, and Gregory D Abowd. 2012. Automatic Assessment of Problem Behavior in Individuals with Developmental Disabilities.
[38]
Thomas Plötz, Paula Moynihan, Cuong Pham, and Patrick Olivier. 2010. Activity Recognition and Healthier Food Preparation. Activity Recognition in Pervasive Intelligent Environments (2010).
[39]
Zhiquan Qi, Bo Wang, Yingjie Tian, and Peng Zhang. 2016. When Ensemble Learning Meets Deep Learning: a New Deep Support Vector Machine for Classification. Knowledge-Based Systems 107 (Sept. 2016), 54--60.
[40]
Nastaran M Rad, Andrea Bizzego, Seyed M Kia, Giuseppe Jurman, Paola Venuti, and Cesare Furlanello. 2015. Convolutional Neural Network for Stereotypical Motor Movement Detection in Autism. In arXiv.1511.01865. Attila Reiss and Didier Stricker. 2012. Introducing a New Benchmarked Dataset for Activity Monitoring.
[41]
Charissa A Ronaoo and Sung-Bae Cho. 2015. Evaluation of deep convolutional neural network architectures for human activity recognition with smartphone sensors. In Proc. KIISE Korea Computer Congress.
[42]
Robert E Schapire. 1990. The strength of weak learnability. Machine Learning (1990), 197--227.
[43]
Albrecht Schmidt, Michael Beigl, and Hans W Gellersen. 1999. There is more to Context than Location. Computer 8 Graphics 23, 6 (1999), 893--901.
[44]
Urminder Singh, Sucheta Chauhan, A. Krishnamachari, and Lovekesh Vig. 2015. Ensemble of Deep Long Short Term Memory Networks for Labelling Origin of Replication Sequences. In Proc. Int. Conf. Data Science and Advanced Analytics (DSAA).
[45]
Yilin Song, Jonathan Viventi, and Yao Wang. 2016. Diversity encouraged learning of unsupervised LSTM ensemble for neural activity video prediction. arXiv preprint 1611.04899v (cs.CV) (2016).
[46]
J. Michael Steele. 2004. The Gauchy-Schwarz Master Class: An Introduction to the Art of Mathematical Inequalities. Cambridge University Press, New York, NY, USA.
[47]
Thomas Stiefmeier, Daniel Roggen, Gerhard Tröster, Georg Ogris, and Paul Lukowicz. 2008. Wearable activity tracking in car manufacturing. IEEE Pervasive Computing 7, 2 (March 2008), 42--50.
[48]
Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints abs/1605.02688 (May 2016). http://arxiv.org/abs/1605.02688
[49]
Giorgio Valentini and Thomas G. Dietterich. 2004. Bias-Variance Analysis of Support Vector Machines for the Development of SVM-Based Ensemble Methods. Journal of Machine Learning Research (JMLR) 5 (2004), 725--775.
[50]
Mark Weiser. 1991. The Computer for the 21st Century. Scientific American (1991).
[51]
Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li 0001, and Shonali Krishnaswamy. 2015. Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition.
[52]
Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, and Joy Zhang. 2014. Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors. Proc. Int. Conf. Mobile Commuting, Applications and Services (MobiCASE) (2014).
[53]
Licheng Zhang, Xihong Wu, and Dingsheng Luo. 2015. Human activity recognition with HMM-DNN model. In Proc. Int. Conf. Cognitive Informatics 8 Cognitive Computing (ICCI). 192--197.
[54]
Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. 2002. Ensembling neural networks: Many could be better than all. Artificial Intelligence 137, 1 (2002), 239--263.
Information & Contributors
Information
Published In
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 2
June 2017
665 pages
Copyright © 2017 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 June 2017
Accepted: 01 May 2017
Revised: 01 March 2017
Received: 01 November 2016
Published in IMWUT Volume 1, Issue 2
Permissions
Request permissions for this article.
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations
- Downloads (Last 12 months)252
- Downloads (Last 6 weeks)45
Reflects downloads up to 16 Nov 2024
Other Metrics
Citations
- Wang CDing YZhou MTang L(2024)Wi-Fi sensing gesture control algorithm based on semi-supervised generative adversarial networkPeerJ Computer Science10.7717/peerj-cs.240210(e2402)Online publication date: 30-Oct-2024
- Haque SDebnath MYasmin AMahmud TNgu A(2024)Experimental Study of Long Short-Term Memory and Transformer Models for Fall Detection on SmartwatchesSensors10.3390/s2419623524:19(6235)Online publication date: 26-Sep-2024
- Cherian JRay STaele PKoh JHammond T(2024)Exploring the Impact of the NULL Class on In-the-Wild Human Activity RecognitionSensors10.3390/s2412389824:12(3898)Online publication date: 16-Jun-2024
- Haresamudram HEssa IPlötz T(2024)Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity RecognitionSensors10.3390/s2404123824:4(1238)Online publication date: 15-Feb-2024
- Kaseris MKostavelis IMalassiotis S(2024)A Comprehensive Survey on Deep Learning Methods in Human Activity RecognitionMachine Learning and Knowledge Extraction10.3390/make60200406:2(842-876)Online publication date: 18-Apr-2024
- Sahoo GQu Y(2024)Improving the Efficiency of Medical Malpractice Attorneys in Collecting, Processing, and Analyzing Data Via a Software-Assisted SolutionEuropean Journal of Electrical Engineering and Computer Science10.24018/ejece.2024.8.1.6008:1(15-22)Online publication date: 12-Feb-2024
- Raja Sekaran SPang YYou LYin O(2024)A hybrid TCN-GRU model for classifying human activities using smartphone inertial signalsPLOS ONE10.1371/journal.pone.030465519:8(e0304655)Online publication date: 13-Aug-2024
- Hwang SLeng ZOh SKim KPlötz TKostakos VKay JHoang T(2024)More Data for People with Disabilities! Comparing Data Collection Efforts for Wheelchair Transportation Mode DetectionProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676617(82-88)Online publication date: 5-Oct-2024
- Hiremath SPlötz TKostakos VKay JHoang T(2024)Game of LLMs: Discovering Structural Constructs in Activities using Large Language ModelsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678444(487-492)Online publication date: 5-Oct-2024
- Arakawa RYakura HGoel M(2024)PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a SmartwatchProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676350(1-16)Online publication date: 13-Oct-2024
- Show More Cited By
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Full Access
View options
View or Download as a PDF file.
eReader
View online with eReader.
Media
Figures
Other
Tables
Affiliations
Yu Guan
Open Lab, Newcastle University, Newcastle upon Tyne, UK
Thomas Plötz
School of Interactive Computing, Georgia Institute of Technology, Atlanta, USA