Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces
View / Open Files
Publication Date
2018-11Journal Title
IEEE/ACM Transactions on Audio Speech and Language Processing
ISSN
2329-9290
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Volume
26
Issue
11
Pages
2083-2097
Type
Article
Metadata
Show full item recordCitation
Weisz, G., Budzianowski, P., Su, P., & Gasic, M. (2018). Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces. IEEE/ACM Transactions on Audio Speech and Language Processing, 26 (11), 2083-2097. https://doi.org/10.1109/TASLP.2018.2851664
Abstract
In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments however are fully observable and have a relatively small action set so in this paper we examine the application of ACER to dialogue policy optimisation. We show that this method beats the current state-of-the-art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state-of-the-art.
Sponsorship
Toshiba Research Europe Ltd, Cambridge Research Laboratory - RG85875
EPSRC Research Council - RG80792
Identifiers
External DOI: https://doi.org/10.1109/TASLP.2018.2851664
This record's URL: https://www.repository.cam.ac.uk/handle/1810/284115
Rights
Licence:
http://www.rioxx.net/licenses/all-rights-reserved
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.