Adaptive packet scheduling in IoT environment based on Q-learning (original) (raw)

Abstract

In the internet of things (IoT) environment consisting of various devices the traffic condition dynamically changes. Failure to process them in complying with the QoS requirement can significantly degrade the reliability and quality of the system. Therefore, the gateway collecting the data needs to quickly establish a new scheduling policy according to the changing traffic condition. The traditional packet scheduling schemes are not effective for IoT since the data transmission pattern is not identified in advance. Q-learning is a type of reinforcement learning that can establish a dynamic scheduling policy without any prior knowledge on the network condition. In this paper a novel Q-learning scheme is proposed which updates the Q-table and reward table based on the condition of the queues in the gateway. Computer simulation reveals that the proposed scheme significantly increases the number of packets satisfying the delay requirement while decreasing the processing time compared to the existing scheme based on Q-learning with stochastic learning automaton. And the processing time is also minimized by omitting unnecessary computation steps in selecting the action in the iterative Q-learning operations.

Access this article

Log in via an institution

Subscribe and save

• Starting from 10 chapters or articles per month
• Access and download chapters and articles from more than 300k books and 2,500 journals
• Cancel anytime View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

• Ahmad S, Malik S, Kim D-H (2018) Comparative analysis of simulation tools with visualization based on real-time task scheduling algorithms for IoT embedded applications. Int J Grid Distrib Comput 11:1–10
  Article Google Scholar
• Alsheikh MA, Hoang DT, Niyato D, Tan H-P, Lin S (2015) Markov decision processes with applications in wireless sensor networks: a survey. IEEE Commun Surv Tutor 17:1239–1267. https://doi.org/10.1109/COMST.2015.2420686
  Article Google Scholar
• Babar M, Arif F (2018) Real-time data processing scheme using big data analytics in internet of things based smart transportation environment. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0820-5
  Article Google Scholar
• Cao B, Li Y, Wang C, Feng G, Oin S, Zhou Y (2017) Resource allocation in software defined wireless networks. IEEE Netw 31:44–51. https://doi.org/10.1109/MNET.2016.1500273NM
  Article Google Scholar
• Chekired DA, Khoukhi L, Mouftah HT (2018) Industrial IoT data scheduling based on hierarchical fog computing: a key for enabling smart factory. IEEE Trans Ind Inf 14:1–12. https://doi.org/10.1109/TII.2018.2843802
  Article Google Scholar
• Chen H, Zhu X, Guo H, Zhu J, Qin X, Wu J (2015) Towards energy-efficient scheduling for real-time tasks under uncertain cloud computing environment. J Syst Softw 99:20–35. https://doi.org/10.1016/j.jss.2014.08.065
  Article Google Scholar
• Desai R, Patil BP (2015) Cooperative reinforcement learning approach for routing in ad hoc networks. In: International conference on pervasive computing, pp 1–5. https://doi.org/10.1109/PERVASIVE.2015.7086962
• Dey N, Hassanien AE, Bhatt C, Ashour AS (2018) Internet of things and big data analytics toward next-generation intelligence. Springer, Berlin. https://doi.org/10.1007/978-3-319-60435-0
  Book Google Scholar
• Diro AA, Reda HT, Chilamkurti N (2018) Differential flow space allocation scheme in SDN based fog computing for IoT applications. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0677-z
  Article Google Scholar
• Dong Z, Liu N, Rojas-Cessa R (2015) Greedy scheduling of tasks with time constraints for energy-efficient cloud-computing data centers. J Cloud Comput Adv Syst Appl. https://doi.org/10.1186/s13677-015-0031-y
  Article Google Scholar
• Ferra H, Lau K, Leckie C, Tang A (2003) Applying reinforcement learning to packet scheduling in routers. In: Proceeding of the fifteenth innovative applications of artificial intelligence conference, pp 79–84
• Hall J, Mars P (1998) Satisfying QoS with a learning based scheduling algorithm. In: 6th international workshop on quality of service, pp 171–173. https://doi.org/10.1109/IWQOS.1998.675236
• He X, Wang K, Huang H, Miyazaki T, Wang Y, Guo S (2018) Green resource allocation based on deep reinforcement learning in content-centric IoT. IEEE Trans Emerg Top Comput. https://doi.org/10.1109/TETC.2018.2805718
  Article Google Scholar
• Itoh N, Kaneko H, Kohiga A, Iwai T, Shimonishi H (2017) Novel packet scheduling for supporting various real-time IoT applications in LTE networks. In: IEEE international workshop technical committee on communications quality and reliability, pp 1–6. http://dx.doi.org/10.1109/CQR.2017.8289445
• Kim D, Lee T, Kim S, Lee B, Youn HY (2018) Adaptive packet scheduling in IoT environment based on Q-learning. In: International conference on emerging ubiquitous systems and pervasive networks. https://doi.org/10.1016/j.procs.2018.10.178
  Article Google Scholar
• Kumar RA, Varshini KM (2014) Multilevel priority packet scheduling scheme for wireless networks. Int J Distrib Parallel Syst 5:69–76
  Google Scholar
• Li L, Li S, Zhao S (2014) QoS-aware scheduling of services-oriented internet of things. IEEE Trans Ind Inform 10:1497–1505. https://doi.org/10.1109/TII.2014.2306782
  Article Google Scholar
• Narman HS, Hossain S, Atizuzzaman M, Schen H (2017) Scheduling internet of things applications in cloud computing. Ann Telecommun 72:79–83. https://doi.org/10.1007/s12243-016-0527-6
  Article Google Scholar
• Nasser N, Karim L, Taleb T (2013) Dynamic multilevel priority packet scheduling scheme for wireless sensor network. IEEE Trans Wirel Commun 12:1448–1459. https://doi.org/10.1109/TWC.2013.021213.111410
  Article Google Scholar
• Orhean AI, Florin Pop, Ioan Raicu (2017) New scheduling approach using reinforcement learning for heterogeneous distributed systems. J Parallel Distrib Comput 117:292–302. https://doi.org/10.1016/j.jpdc.2017.05.001
  Article Google Scholar
• Peng Z, Cui D, Zuo J, Li Q, Xu B, Lin W (2015) Random task scheduling based on reinforcement learning in cloud computing. Cluster Comput 18:1595–1607. https://doi.org/10.1007/s10586-015-0484-2
  Article Google Scholar
• Shahrabi J, Adibi MA, Mahootchi M (2017) A reinforcement learning approach to parameter estimation in dynamic job shop scheduling. Comput Ind Eng 110:75–82. https://doi.org/10.1016/j.cie.2017.05.026
  Article Google Scholar
• Sharma R, Kumar N, Gowda NB, Srinivas T (2018) Packet scheduling scheme to guarantee QoS in internet of things. Wirel Pers Commun 100:557–569. https://doi.org/10.1007/s11277-017-5218-8
  Article Google Scholar
• Sledge IJ, Principe JC (2017) Balancing Exploration and Exploitation in Reinforcement Learning using a Value of Information Criterion. In: IEEE international conference on acoustics, speech and signal processing, pp 2816–2820. https://doi.org/10.1109/ICASSP.2017.7952670
• Sombolestan SM, Rasooli A, Khodaygan (2018) Optimal path-planning for mobile robots to find a hidden target in an unknown environment based on machine learning. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0777-4
  Article Google Scholar
• Tijsma AD, Drugan MM, Wiering MA (2016) Comparing Exploration Strategies for Q-learning in Random Stochastic Mazes. In: IEEE Symposium series on computational intelligence, pp 1–8
• Yau K-LA, Komisarczuk P, Teal PD (2012) Reinforcement learning for context awareness and intelligence in wireless networks: review, new features and open issues. J Netw Comput Appl 35:253–267. https://doi.org/10.1016/jjnca.2011.08.007
  Article Google Scholar
• Zhang J, Liu X (2018) Evaluation and optimization of QoS-aware network management framework based on process synergy and resource allocation. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0689-3
  Article Google Scholar

Download references

Acknowledgements

This work was partly supported by Institute for Information and Communications Technology Promotion (IITP) Grant funded by the Korea government(MSIT) (No. 2016-0-00133, Research on Edge computing via collective intelligence of hyperconnection IoT nodes), Korea, under the National Program for Excellence in SW supervised by the IITP (Institute for Information and communications Technology Promotion) (2015-0-00914), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2016R1A6A3A119 31385, Research of key technologies based on software defined wireless sensor network for realtime public safety service, 2017R1A2B2009095, Research on SDN-based WSN Supporting Real-time Stream Data Processing and Multiconnectivity), the second Brain Korea 21 PLUS project.

Author information

Authors and Affiliations

1. Department of Electronic, Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 440746, South Korea
   Donghyun Kim, Taeho Lee, Sejun Kim, Byungjun Lee & Hee Yong Youn

Authors

1. Donghyun Kim
2. Taeho Lee
3. Sejun Kim
4. Byungjun Lee
5. Hee Yong Youn

Corresponding author

Correspondence toHee Yong Youn.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

About this article

Cite this article

Kim, D., Lee, T., Kim, S. et al. Adaptive packet scheduling in IoT environment based on Q-learning.J Ambient Intell Human Comput 11, 2225–2235 (2020). https://doi.org/10.1007/s12652-019-01351-w

Download citation

• Received: 02 January 2019
• Accepted: 05 May 2019
• Published: 18 June 2019
• Version of record: 18 June 2019
• Issue date: June 2020
• DOI: https://doi.org/10.1007/s12652-019-01351-w

Keywords