Finding misplaced items using a mobile robot in a smart home environment (original) (raw)

Abstract

Smart homes can provide complementary information to assist home service robots. We present a robotic misplaced item finding (MIF) system, which uses human historical trajectory data obtained in a smart home environment. First, a multi-sensor fusion method is developed to localize and track a resident. Second, a path-planning method is developed to generate the robot movement plan, which considers the knowledge of the human historical trajectory. Third, a real-time object detector based on a convolutional neural network is applied to detect the misplaced item. We present MIF experiments in a smart home testbed and the experimental results verify the accuracy and efficiency of our solution.

Similar content being viewed by others

References

• Ahern SJ, Carter J, Wilson P, 2015. Autonomous robot retrieval system. SAI Intelligent Systems Conf, p.280–282. https://doi.org/10.1109/IntelliSys.2015.7361155
  Google Scholar
• Castellanos JA, Neira J, Tardós JD, 2001. Multisensor fusion for simultaneous localization and map building. IEEE Trans Rob Autom, 17(6):908–914. https://doi.org/10.1109/70.976024
  Article Google Scholar
• Choi YS, Deyle T, Chen T, et al., 2009. A list of household objects for robotic retrieval prioritized by people with ALS. IEEE Int Conf on Rehabilitation Robotics, p.510–517. https://doi.org/10.1109/ICORR.2009.5209484
  Google Scholar
• Ekvall S, Kragic D, Jensfelt P, 2007. Object detection and mapping for service robot tasks. Robotica, 25(2):175–187. https://doi.org/10.1017/S0263574706003237
  Article Google Scholar
• Fujii T, Ueda H, Minoh M, 2007. A looking-for-objects service in ubiquitous home. J Nat Inst Inform Commun Technol, 54(3):175–181.
  Google Scholar
• Huynh SM, Parry D, Fong ACM, et al., 2014. Home localization system for misplaced objects. IEEE Int Conf on Consumer Electronics, p.462-463. https://doi.org/10.1109/ICCE.2014.6776087
  Book Google Scholar
• Kollar T, Roy N, 2009. Utilizing object-object and object-scene context when planning to find things. IEEE Int Conf on Robotics and Automation, p.2168–2173. https://doi.org/10.1109/ROBOT.2009.5152831
  Google Scholar
• Komatsuzaki M, Tsukada K, Siio I, et al., 2011. IteMinder: finding items in a room using passive RFID tags and an autonomous robot (poster). Proc 13th Int Conf on Ubiquitous Computing, p.599–600. https://doi.org/10.1145/2030112.2030232
  Google Scholar
• Lin YC, Wei ST, Yang SA, et al., 2015. Planning on searching occluded target object with a mobile robot manipulator. IEEE Int Conf on Robotics and Automation, p.3110–3115. https://doi.org/10.1109/ICRA.2015.7139626
  Google Scholar
• Meng Q, Lee MH, 2006. Design issues for assistive robotics for the elderly. Adv Eng Inform, 20(2):171–186. https://doi.org/10.1016/j.aei.2005.10.003
  Article Google Scholar
• PIR, 2018. Passive Infrared Sensor. https://en.wikipedia.org/wiki/Passive\_infrared\_sensor
  Google Scholar
• Ravi K, Bhavani KJ, Vinayak M, et al., 2016. Identification of misplaced objects. Indian J Sci Technol, 9(17):1–4. https://doi.org/10.17485/ijst/2016/v9i17/92991
  Article Google Scholar
• Redmon J, Divvala S, Girshick R, et al., 2016. You only look once: unified, real-time object detection. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.779–788. https://doi.org/10.1109/CVPR.2016.91
  Google Scholar
• Rogers JG, Christensen HI, 2013. Robot planning with a semantic map. IEEE Int Conf on Robotics and Automation, p.2239-2244. https://doi.org/10.1109/ICRA.2013.6630879
  Book Google Scholar
• ROS, 2018. Robot Operating System. http://www.ros.org/
  Google Scholar
• Sadhu C, Abhiram MH, Chandan B, et al., 2013. Cognitive learning enabled real time object search robot. Int Conf on Control, Automation, Robotics and Embedded System, p.1–6. https://doi.org/10.1109/CARE.2013.6733739
  Google Scholar
• Schwarz D, Schwarz M, Stückler J, et al., 2014. Cosero, find my keys! Object localization and retrieval using bluetooth low energy tags. Proc 18th RoboCup Int Symp, p.195–206. https://doi.org/10.1007/978-3-319-18615-3\_16
  Google Scholar
• Susi M, Renaudin V, Lachapelle G, 2013. Motion mode recognition and step detection algorithms for mobile phone users. Sensors, 13(2):1539–1562. https://doi.org/10.3390/s130201539
  Article Google Scholar
• Tesoriero R, Gallud JA, Lozano MD, et al., 2009. Tracking autonomous entities using RFID technology. IEEE Trans Consum Electron, 55(2):650–655. https://doi.org/10.1109/TCE.2009.5174435
  Article Google Scholar
• Wang Q, 2018. Youtube Video on Robotic Misplaced Item Finding System. https://youtu.be/jEDKbr71pis
  Google Scholar
• Wang Q, Zhang SL, Liu MQ, et al., 2017. Retrieval of misplaced items using a mobile robot via visual object recognition. IEEE 7th Conf on CYBER Technology in Automation, Control, and Intelligent Systems, p.1188–1193. https://doi.org/10.1109/CYBER.2017.8446158
  Google Scholar
• Xu D, Huang Q, Liu HW, 2016. Object detection on robot operation system. IEEE 11th Conf on Industrial Electronics and Applications, p.1155–1159. https://doi.org/10.1109/ICIEA.2016.7603758
  Google Scholar
• Xunfei, 2018. Voice Dictation. http://www.xfyun.cn/services/voicedictation
  Google Scholar

Download references

Author information

Authors and Affiliations

1. College of Electrical Engineering, Zhejiang University, Hangzhou, 310027, China
   Qi Wang, Zhen Fan, Sen-lin Zhang & Mei-qin Liu
2. School of Electrical and Computer Engineering, Oklahoma State University, Stillwater, OK, 74078, USA
   Wei-hua Sheng

Authors

1. Qi Wang
2. Zhen Fan
3. Wei-hua Sheng
4. Sen-lin Zhang
5. Mei-qin Liu

Corresponding author

Correspondence toMei-qin Liu.

Ethics declarations

Qi WANG, Zhen FAN, Wei-hua SHENG, Sen-lin ZHANG, and Mei-qin LIU declare that they have no conflict of interest.

Additional information

Project supported by the Basic Public Research Program of Zhejiang Province, China (No. LGF18F030001) and the Open Research Project of the State Key Laboratory of Industrial Control Technology, Zhejiang University, China (No. ICT1800414)

Rights and permissions

About this article

Cite this article

Wang, Q., Fan, Z., Sheng, Wh. et al. Finding misplaced items using a mobile robot in a smart home environment.Frontiers Inf Technol Electronic Eng 20, 1036–1048 (2019). https://doi.org/10.1631/FITEE.1800275

Download citation

• Received: 03 May 2018
• Accepted: 30 July 2018
• Published: 02 September 2019
• Version of record: 02 September 2019
• Issue date: August 2019
• DOI: https://doi.org/10.1631/FITEE.1800275

Key words

CLC number