Xunying Liu - Academia.edu (original) (raw)
Related Authors
Université Libre De Tunis
Uploads
Papers by Xunying Liu
Recurrent neural network language models (RNNLMs) are be coming increasingly popular for speech r... more Recurrent neural network language models (RNNLMs) are be coming increasingly popular for speech recognition. Previously, we have shown that RNNLMs with a full (non-classed) output layer (F-RNNLMs) can be trained efficiently using a GPU giving a large reduction in training time over conventional class-based models (C -RNNLMs) on a standard CPU. However, since test-time RNNLM evaluation is often performed entirely on a CPU, standard F-RNNLMs are inefficient since the entire output layer needs to be calculated for normalisation. In this paper, it is demonstrated that C-RNNLMs can be efficiently trained on a GPU, using our spliced sentence bunch technique which allows good CPU test-time performance (42x speedup over F-RNNLM). Furthermore, the per formance of different classing approaches is investigated. We also examine the use of variance regularisation of the softmax denom inator for F-RNNLMs and show that it allows F-RNNLMs to be efficiently used in test (56x speedup on a CPU). Finally the use of two GPUs for F-RNNLM training using pipelining is described and shown to give a reduction in training time over a single GPU by a factor of 1.6 x.
Recurrent neural network language models (RNNLMs) are be coming increasingly popular for speech r... more Recurrent neural network language models (RNNLMs) are be coming increasingly popular for speech recognition. Previously, we have shown that RNNLMs with a full (non-classed) output layer (F-RNNLMs) can be trained efficiently using a GPU giving a large reduction in training time over conventional class-based models (C -RNNLMs) on a standard CPU. However, since test-time RNNLM evaluation is often performed entirely on a CPU, standard F-RNNLMs are inefficient since the entire output layer needs to be calculated for normalisation. In this paper, it is demonstrated that C-RNNLMs can be efficiently trained on a GPU, using our spliced sentence bunch technique which allows good CPU test-time performance (42x speedup over F-RNNLM). Furthermore, the per formance of different classing approaches is investigated. We also examine the use of variance regularisation of the softmax denom inator for F-RNNLMs and show that it allows F-RNNLMs to be efficiently used in test (56x speedup on a CPU). Finally the use of two GPUs for F-RNNLM training using pipelining is described and shown to give a reduction in training time over a single GPU by a factor of 1.6 x.