A hybrid framework for smile detection in class imbalance scenarios (original) (raw)

Abstract

In this study, we consider the problem of smile detection in both an imbalanced data scenario, in which the number of smile images is in the minority compared with the number of neutral images, and a balanced data scenario. We first propose a smile detection model using a convolutional neural network (SD-CNN) to improve the performance in the balanced data scenario, and then a hybrid deep learning framework (HF-SD) that uses a modification of the SD-CNN model to learn and then extracts the features from dataset. These extracted features are then used to train an extreme gradient boosting approach to handle the imbalanced problem. An experiment shows that the proposed model has impressive discriminative ability for smile detection, in both balanced and imbalanced data scenarios, compared with existing approaches. HF-SD yields an accuracy of 93.6% and outperforms the state-of-the-art approaches for the original GENK14K database in the balanced data scenario. The results of the second experiment show that HF-SD also achieves better AUCs (area under the receiver operating characteristic curve) compared with the state-of-the-art methods for smile detection in an imbalanced data scenario with different balancing ratios.

Access this article

Log in via an institution

Subscribe and save

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Ali M, Le HS, Khan M, Nguyen TT (2018) Segmentation of dental X-ray images in medical imaging using neutrosophic orthogonal matrices. Expert Syst Appl 91:434–441
    Article Google Scholar
  2. An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363
    Article Google Scholar
  3. Bianco S, Celona L, Schettini R (2016) Robust smile detection using convolutional neural networks. J Electron Imaging 25(6):063002
    Article Google Scholar
  4. Chen J, Ou Q, Chi Z, Fu H (2017) Smile detection in the wild with deep convolutional neural networks. Mach Vis Appl 28(1–2):173–183
    Article Google Scholar
  5. Chen T, Guestrin T (2016) XGBoost: a scalable tree boosting system. In: KDD, pp 785–794
  6. Chen Y, Zou W, Tang Y, Li X, Xu C, Komodakis N (2018) SCOM: spatiotemporal constrained optimization for salient object detection. IEEE Trans Image Process 27(7):3345–3357
    Article MathSciNet Google Scholar
  7. Dinh VS, Le TBC, Do PT (2017) Facial smile detection using convolutional neural networks. In: KSE’17, pp 136–141
  8. Galar M, Fernández A, Tartas EB, Sola HB, Herrera F (2012) A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans Syst Man Cybern Part C 42(4):463–484
    Article Google Scholar
  9. Gao Y, Liu H, Wu P, Wang C (2016) A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios. Neurocomputing 174:1077–1086
    Article Google Scholar
  10. Han D, Liu Q, Fan W (2018) A new image classification method using CNN transfer learning and web data augmentation. Expert Syst Appl 95:43–56
    Article Google Scholar
  11. He H, Garcia EA (2009) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284
    Article Google Scholar
  12. Hu L, Ni Q (2018) IoT-driven automated object detection algorithm for urban surveillance systems in smart cities. IEEE Internet Things J 5(2):747–754
    Article Google Scholar
  13. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML’15, pp 448–456
  14. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758
    Google Scholar
  15. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations (ICLR)
  16. Le A, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363
    Article Google Scholar
  17. Le HS, Pham HT (2017) Some novel hybrid forecast methods based on picture fuzzy clustering for weather nowcasting from satellite image sequences. Appl Intell 46(1):1–15
    Article MathSciNet Google Scholar
  18. Le HS, Tran MT, Fujita H, Dey N, Ashour AS, Vo TNN, Le QA, Chu DT (2018) Dental diagnosis from x-ray images: an expert system based on fuzzy computing. Biomed Signal Process Control 39:64–73
    Article Google Scholar
  19. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444
    Article Google Scholar
  20. Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: CVPR, pp 5325–5334
  21. Liu M, Li S, Shan S, Chen X (2012) Enhancing expression recognition in the wild with unlabeled reference data. In: Asian conference on computer vision (ACCV), pp 577–588
    Chapter Google Scholar
  22. Lopes AT, Aguiar E, Souza AFD, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628
    Article Google Scholar
  23. Parka BH, Oha SY, Kim IJ (2017) Face alignment using a deep neural network with local feature learning and recurrent regression. Expert Syst Appl 89:66–80
    Article Google Scholar
  24. Peng H, Li B, Ling H, Hu W, Xiong W, Maybank SJ (2017) Salient object detection via structured matrix decomposition. IEEE Trans Pattern Anal Mach Intell 39(4):818–832
    Article Google Scholar
  25. Raza M, Chen Z, Rehman SU, Wang P, Bao P (2018) Appearance based pedestrians’ head pose and body orientation estimation using deep learning. Neurocomputing 272:647–659
    Article Google Scholar
  26. Ren S, He K, Girshick RB, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149
    Article Google Scholar
  27. Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018
    Article MathSciNet Google Scholar
  28. Shan C (2012) Smile detection by boosting pixel differences. IEEE Trans Image Process 21(1):431–436
    Article MathSciNet Google Scholar
  29. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR
  30. Vo T, Nguyen T, Le CT (2018) Race recognition using deep convolutional neural networks. Symmetry 10(11):564
    Article Google Scholar
  31. Wei Y, Xia W, Lin M, Huang J, Ni B, Dong J, Zhao Y, Yan S (2016) HCP: a flexible CNN framework for multi-label image classification. IEEE Trans Pattern Anal Mach Intell 38(9):1901–1907
    Article Google Scholar
  32. Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Toward practical smile detection. IEEE Trans Pattern Anal Mach Intell 31:2106–2111
    Article Google Scholar
  33. Zhang Y, Zhang E, Chen W (2016) Deep neural network for halftone image classification based on sparse auto-encoder. Eng Appl Artif Intell 50:245–255
    Article Google Scholar

Download references

Author information

Authors and Affiliations

  1. Institute of Research and Development, Duy Tan University, Da Nang, 550000, Vietnam
    Thanh Vo
  2. Faculty of Information Technology, Ho Chi Minh City Open University, Ho Chi Minh City, Vietnam
    Trang Nguyen
  3. Faculty of Information Technology, Ho Chi Minh City University of Technology (HUTECH), Ho Chi Minh City, Vietnam
    C. T. Le

Authors

  1. Thanh Vo
  2. Trang Nguyen
  3. C. T. Le

Corresponding author

Correspondence toC. T. Le.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

About this article

Cite this article

Vo, T., Nguyen, T. & Le, C.T. A hybrid framework for smile detection in class imbalance scenarios.Neural Comput & Applic 31, 8583–8592 (2019). https://doi.org/10.1007/s00521-019-04089-w

Download citation

Keywords