A Survey on Reinforcement Learning for Dialogue Systems (original) (raw)
Abstract
Dialogue systems are computer systems which com- municate with humans using natural language. The goal is not just to imitate human communication but to learn from these interactions and improve the system’s behaviour over time. Therefore, different machine learning approaches can be implemented with Reinforcement Learning being one of the most promising techniques to generate a contextually and semantically appropriate response. This paper outlines the current state-of- the-art methods and algorithms for integration of Reinforcement Learning techniques into dialogue systems.
Figures (6)
Fig. 1. The rise of DS illustrated by the search frequency of the term chatbot on Google Trends worldwide. The values indicate the search interest relative to the highest point on the graph in the specified time period, whereby the value 100 stands for the highest popularity of this search term. Google Trends: https://trends.google.de/trends/explore?date=all&q=chatbot (accessed on 15.01.19) DS become more and more important in society with humans interacting every day with personal assistants like Siri, Google Now, Cortana and Alexa — a fact which can also be illustrated by people’s search behaviour on Google (Fig. 1). In March 2016 at Microsoft Build 2016, Microsoft CEO Satya Nadella introduced the term conversation as a platform (CaaP) to fa- cilitate the creation of even more advanced personal assistants.
Fig. 3. Simplified general architecture of a DS, combining the most common properties from the different architectures of DS. The dialogue can be modelle as an RL-problem where the DS is the RL agent and the user is the environment; based on [1] [8] [9].
‘ig. 5. A classification scheme of DS categorized by ordered pairs of the -onversation ability on the ordinate and the response mechanism on the ibscissa whereby DS can be based on rules or artificial intelligence (AI). A retrieval-based DS in an open domain is not possible whereas a DS which etrieves responses for a predefined topic domain is the comparatively simplest ‘orm of a DS. If the DS is able to create new responses in a closed domain, it is egarded as a weak AI. In contrast, a generative-based DS without limitations yf a conversational domain would have true intelligence.
Fig. 6. The dialogue model and policy model form the centre of the task- oriented system with belief state tracking and RL. In contrast to the MDP, the input utterance is regarded as an observation of the underlying user intent which is hidden. Instead of trying to estimate the hidden dialogue state, the system response is directly given from the distribution over all possible dialogue states [12]. The policy model determines which action to take at each turn. As the dialogue progresses, a reward is given at each step to reflect the desired characteristics of the DS. The dialogue model M and policy model P can be optimized using RL from these rewards either through interaction with users or from a corpus of dialogues (Fig. 6 on the previous page) [12].
Fig. 7. Model of the DS where a knowledge transfer can be realised due to a domain overlap of the source domain MovieBooking and the target domain RestaurantBooking, both sharing the same information date [17]. CEEIC UU ULI SS Lit Pecui dit, The approach of Ilievski et al. (2018) [17] also addresses the Ss ing e-domain problem as they introduced a goal-oriented DS using a transfer learning method based on earlier work of [ inde t ne ne need transfer learning technique can also be applied if a third d 13] and [18]’. As both approaches implemented bots pendently for their respective domain, [17] are utilising similarities between a source and target domain and transferring the knowledge from one neural network to anot her. As illustrated in Fig. 7, two domains can include same type of information and therefore, there is no for learning this particular information twice. This omain, for example, the domain Tourism, is added which shares all information from the source domain and came additinnal information like tyne af accqmmoadatiann [171]
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.
References (19)
- Jurafsky, D. and Martin, J. H. (2018): Speech and Language Process- ing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice-Hall. Draft of September 23, 2018. [online] https://web.stanford.edu/ ∼ jurafsky/slp3/ed3book.pdf (accessed on 15.01.19)
- Sutton, R. S., & Barto, A. G. (1998): Introduction to reinforcement learning (Vol. 135). Cambridge: MIT press.
- Müller-Schloer, C., & Tomforde, S. (2017): Organic Computing- Technical Systems for Survival in the Real World. Springer International Publishing.
- Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M. (2016): Mastering the game of go with deep neural networks and tree search. Nature, 529(7587),484-489.
- Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996): Reinforcement learning: A survey. Journal of artificial intelligence research, 4, 237-285.
- Lai, M. (2015): Giraffe: Using deep reinforcement learning to play chess. arXiv preprint arXiv:1509.01549.
- Epshteyn, A., Vogel, A., & DeJong, G. (2008): Active reinforcement learning. In Proceedings of the 25th international conference on Machine learning, 296-303. ACM.
- Singh, M. (2004): Practical Handbook of Internet Computing. Chapman and Hall/CRC.
- Petraityt, J. (2018): Deprecating the state machine: building conversa- tional AI with the Rasa stack, PyData Berlin 2018.
- Martin-Martin, A. (2017): Can we use Google Scholar to identify highly- cited documents?
- Weizenbaum, J. (1983): ELIZA -A Computer Program for the Study of Natural Language Communication Between Man and Machine, Communications of the ACM, vol. 26, no. 1, 23-28.
- Young, S., Gai, M., Thomson, B., & Williams, J. D. (2013): Pomdp- based statistical spoken dialogue systems: A review. Proceedings of the IEEE, 101(5), 1160-1179.
- Li, J., Monroe, W., Ritter, A., Galley, M., Gao, J., & Jurafsky, D. (2017): Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541.
- Williams, J. D., & Zweig, G. (2016): End-to-end lstm-based dialogue control optimized with supervised and reinforcement learning. arXiv preprint arXiv:1606.01269.
- Su, P. H., Vandyke, D., Gasic, M., Kim, D., Mrksic, N., Wen, T. H., & Young, S. (2015): Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. arXiv preprint arXiv:1508.03386.
- Peng, B., Li, X., Li, L., Gao, J., Celikyilmaz, A., Lee, S., & Wong, K. F. (2017): Composite task-completion dialogue policy learning via hier- archical deep reinforcement learning. arXiv preprint arXiv:1704.03084.
- Ilievski, I., Musat, C., Hossmann, A., & Baeriswyl, M. (2018): Goal- Oriented Chatbot Dialog Management Bootstrapping with Transfer Learning. arXiv preprint arXiv:1802.00500.
- Wen, T., Vandyke, D., Mrksic, N., Gasic, M., Rojas-Barahona, L., Su, P., Ultes, S. & Young, S. (2016): A network-based end-to-end trainable task-oriented dialogue system.arXiv preprint arXiv:1604.04562.
- Serban, I. V., Sankar, C., Germain, M., Zhang, S., Lin, Z., Subramanian, S. & Rajeswar, S. (2018): A Deep Reinforcement Learning Chatbot (Short Version). arXiv preprint arXiv:1801.06700.