A deep Q-learning network based active object detection model with a novel training algorithm for service robots (original) (raw)

Abstract

This paper focuses on the problem of active object detection (AOD). AOD is important for service robots to complete tasks in the family environment, and leads robots to approach the target object by taking appropriate moving actions. Most of the current AOD methods are based on reinforcement learning with low training efficiency and testing accuracy. Therefore, an AOD model based on a deep Q-learning network (DQN) with a novel training algorithm is proposed in this paper. The DQN model is designed to fit the Q-values of various actions, and includes state space, feature extraction, and a multilayer perceptron. In contrast to existing research, a novel training algorithm based on memory is designed for the proposed DQN model to improve training efficiency and testing accuracy. In addition, a method of generating the end state is presented to judge when to stop the AOD task during the training process. Sufficient comparison experiments and ablation studies are performed based on an AOD dataset, proving that the presented method has better performance than the comparable methods and that the proposed training algorithm is more effective than the raw training algorithm.

摘要

本文研究了主动物品检测(AOD)问题。AOD是服务机器人在家庭环境中完成服务任务的重要组成部分,通过适当的移动动作引导机器人接近目标物品。目前基于强化学习的AOD模型存在训练效率低和测试精度差的问题。因此,本文提出一种基于深度Q学习网络的AOD模型,并设计了一种新的模型训练算法。该模型旨在拟合各种动作Q值,包括状态空间、特征提取和多层感知机。与现有研究不同,本文针对所提AOD模型设计了一种基于记忆的训练算法,以提高模型训练效率和测试精度。此外,提出一种最终状态生成方法判断训练过程中AOD任务何时停止。本文所提方法在AOD数据集上进行了充分的对比实验和消融实验。实验结果表明所提方法优于其他同类方法,所设计的训练算法比原始训练算法更高效。

Similar content being viewed by others

References

• Ammirato P, Poirson P, Park E, et al., 2017. A dataset for developing and benchmarking active vision. Proc IEEE Int Conf on Robotics and Automation, p.1378–1385. https://doi.org/10.1109/ICRA.2017.7989164
• Ammirato P, Berg AC, Košecká J, 2018. Active vision dataset benchmark. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops, p.2046–2049. https://doi.org/10.1109/CVPRW.2018.00277
• Dos Reis DH, Welfer D, De Souza Leite Cuadros MA, et al., 2019. Mobile robot navigation using an object recognition software with RGBD images and the YOLO algorithm. Appl Artif Intell, 33(14):1290–1305. https://doi.org/10.1080/08839514.2019.1684778
  Article Google Scholar
• Duan KW, Bai S, Xie LX, et al., 2019. CenterNet: keypoint triplets for object detection. Proc IEEE/CVF Int Conf on Computer Vision, p.6568–6577. https://doi.org/10.1109/ICCV.2019.00667
• Han XN, Liu HP, Sun FC, et al., 2019. Active object detection with multistep action prediction using deep Q-network. IEEE Trans Ind Inform, 15(6):3723–3731. https://doi.org/10.1109/TII.2019.2890849
  Article Google Scholar
• He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.770–778. https://doi.org/10.1109/CVPR.2016.90
• Liu SP, Tian GH, Zhang Y, et al., 2022a. Active object detection based on a novel deep Q-learning network and long-term learning strategy for the service robot. IEEE Trans Ind Electron, 69(6):5984–5993. https://doi.org/10.1109/TIE.2021.3090707
  Article Google Scholar
• Liu SP, Tian GH, Zhang Y, et al., 2022b. Service planning oriented efficient object search: a knowledge-based framework for home service robot. Exp Syst Appl, 187:115853. https://doi.org/10.1016/j.eswa.2021.115853
  Article Google Scholar
• Mnih V, Kavukcuoglu K, Silver D, et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533. https://doi.org/10.1038/nature14236
  Article Google Scholar
• Mousavian A, Toshev A, Fišer M, et al., 2019. Visual representations for semantic target driven navigation. Proc IEEE Int Conf on Robotics and Automation, p.8846–8852. https://doi.org/10.1109/ICRA.2019.8793493
• Paletta L, Pinz A, 2000. Active object recognition by view integration and reinforcement learning. Robot Autom Syst, 31(1–2):71–86. https://doi.org/10.1016/S0921-8890(99)00079-2
  Article Google Scholar
• Pu SL, Zhao W, Chen WJ, et al., 2021. Unsupervised object detection with scene-adaptive concept learning. Front Inform Technol Electron Eng, 22(5):638–651. https://doi.org/10.1631/FITEE.2000567
  Article Google Scholar
• Ren SQ, He KM, Girshick R, et al., 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Patt Anal Mach Intell, 39(6):1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031
  Article Google Scholar
• Schmid JF, Lauri M, Frintrop S, 2019. Explore, approach, and terminate: evaluating subtasks in active visual object search based on deep reinforcement learning. Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems, p.5008–5013. https://doi.org/10.1109/IROS40897.2019.8967805
• Shuai W, Chen XP, 2019. KeJia: towards an autonomous service robot with tolerance of unexpected environmental changes. Front Inform Technol Electron Eng, 20(3):307–317. https://doi.org/10.1631/FITEE.1900096
  Article Google Scholar
• Singh A, Sha J, Narayan KS, et al., 2014. BigBIRD: a large-scale 3D database of object instances. Proc IEEE Int Conf on Robotics and Automation, p.509–516. https://doi.org/10.1109/ICRA.2014.6906903
• van Hasselt H, Guez A, Silver D, 2016. Deep reinforcement learning with double Q-learning. Proc AAAI Conf on Artificial Intelligence, p.2094–2100. https://doi.org/10.1609/aaai.v30i1.10295
• Wan SH, Goudos S, 2020. Faster R-CNN for multi-class fruit detection using a robotic vision system. Comput Netw, 168:107036. https://doi.org/10.1016/j.comnet.2019.107036
  Article Google Scholar
• Wang Q, Fan Z, Sheng WH, et al., 2019. Finding misplaced items using a mobile robot in a smart home environment. Front Inform Technol Electron Eng, 20(8):1036–1048. https://doi.org/10.1631/FITEE.1800275
  Article Google Scholar
• Xu QL, Fang F, Gauthier N, et al., 2021. Towards efficient multiview object detection with adaptive action prediction. Proc IEEE Int Conf on Robotics and Automation, p.13423–13429. https://doi.org/10.1109/ICRA48506.2021.9561388
• Zhang H, Liu HP, Guo D, et al., 2017. From foot to head: active face finding using deep Q-learning. Proc IEEE Int Conf on Image Processing, p.1862–1866. https://doi.org/10.1109/ICIP.2017.8296604
• Zhou XY, Zhuo JC, Krähenbühl P, 2019. Bottom-up object detection by grouping extreme and center points. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.850–859. https://doi.org/10.1109/CVPR.2019.00094

Download references

Author information

Authors and Affiliations

1. School of Control Science and Engineering, Shandong University, Jinan, 250061, China
   Shaopeng Liu (刘少鹏), Guohui Tian (田国会), Yongcheng Cui (崔永成) & Xuyang Shao (邵旭阳)

Authors

1. Shaopeng Liu (刘少鹏)
2. Guohui Tian (田国会)
3. Yongcheng Cui (崔永成)
4. Xuyang Shao (邵旭阳)

Contributions

Shaopeng LIU and Guohui TIAN designed the research. Shaopeng LIU addressed the problems, processed the data, and drafted the paper. Guohui TIAN, Yongcheng CUI, and Xuyang SHAO helped organize the paper. Shaopeng LIU and Guohui TIAN revised and finalized the paper.

Corresponding author

Correspondence toGuohui Tian (田国会).

Additional information

Compliance with ethics guidelines

Shaopeng LIU, Guohui TIAN, Yongcheng CUI, and Xuyang SHAO declare that they have no conflict of interest.

Project supported by the National Natural Science Foundation of China (Nos. U1813215 and 62273203)

Rights and permissions

About this article

Cite this article

Liu, S., Tian, G., Cui, Y. et al. A deep Q-learning network based active object detection model with a novel training algorithm for service robots.Front Inform Technol Electron Eng 23, 1673–1683 (2022). https://doi.org/10.1631/FITEE.2200109

Download citation

• Received: 20 March 2022
• Accepted: 29 July 2022
• Published: 24 September 2022
• Version of record: 24 September 2022
• Issue date: November 2022
• DOI: https://doi.org/10.1631/FITEE.2200109

Key words

关键词

CLC number