Improved Transformer-Based Implicit Latent GAN with Multi-headed Self-attention for Unconditional Text Generation (original) (raw)

Abstract

Generative Adversarial Network (GAN) is widely used in computer vision, such as image generation and other tasks. In recent years, GAN has also been developed in the field of unconditional text generation. In this work, we improve TILGAN for unconditional text generation by refactoring the generator. In short, we use Multi-headed Self-attention to replace the Linear layer and BN layer to endow the generator with better text generation capabilities. Our model consists of three components: a transformer autoencoder, a Multi-headed Self attention based generator and a linear based discriminator. The encoder in transformer autoencoder is used to generate the distribution of real samples, and the decoder is used to decode real or generated sentence vector into text. The loss functions for autoencoder and GAN are cross entropy and KL divergence, respectively. On the MS COCO dataset, the proposed model has achieved a better BLEU score than TILGAN. Our ablation experiments also proved the effectiveness of the proposed generator network for unconditional text generation.

Similar content being viewed by others

References

  1. Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63, 139–144 (2020)
    Article Google Scholar
  2. Kusner, M.J., Hernández-Lobato, J.M.: GANs for sequences of discrete elements with the gumbel-softmax distribution. arXiv arXiv:1611.04051 (2016)
  3. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv arXiv:1701.07875 (2017)
  4. Diao, S., Shen, X., Shum, K., et al.: TILGAN: transformer-based implicit latent GAN for diverse and coherent text generation. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4844–4858 (2021)
    Google Scholar
  5. Nie, W., Narodytska, N., Patel, A.: RelGAN: relational generative adversarial networks for text generation. In: Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada, 30 April–3 May 2018
    Google Scholar
  6. Yu, L., Zhang, W., Wang, J., Yu, Y.: SeqGAN: sequence generative adversarial nets with policy gradient. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017
    Google Scholar
  7. Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., Wang, J.: Long text generation via adversarial training with leaked information. arXiv arXiv:1709.08624 (2017)
  8. Juefei-Xu, F., Dey, R., Boddeti, V.N., Savvides, M.: RankGAN: a maximum margin ranking GAN for generating faces. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 3–18. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20893-6_1
    Chapter Google Scholar
  9. Fedus, W., Goodfellow, I., Dai, A.M.: MaskGAN: better text generation via filling in the ____. arXiv arXiv:1801.07736 (2018)
  10. Liu, Z., Wang, J., Liang, Z.: CatGAN: category-aware generative adversarial networks with hierarchical evolutionary learning for category text generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 8425–8432 (2020)
    Google Scholar
  11. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv arXiv:1412.6980 (2014)
  12. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA, 7–12 July 2002
    Google Scholar
  13. Zhu, Y., et al.: Texygen: a benchmarking platform for text generation models. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097–1100 (2018)
    Google Scholar
  14. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
    Chapter Google Scholar
  15. Chen, L., et al.: Adversarial text generation via feature mover’s distance. In: Advances in Neural Information Processing Systems, pp. 4666–4677 (2018)
    Google Scholar
  16. Wu, H.Y., Chen, Y.L.: Graph sparsification with generative adversarial network. In: 2020 IEEE International Conference on Data Mining (ICDM), pp. 1328–1333. IEEE (2020)
    Google Scholar

Download references

Acknowledgments

This research has been supported by JSPS KAKENHI Grant Number 19K20345.

Author information

Authors and Affiliations

  1. University of Electronic Science and Technology of China, Chengdu, China
    Fuji Ren
  2. Tokushima University, Tokushima, Japan
    Ziyun Jiao & Xin Kang

Authors

  1. Fuji Ren
  2. Ziyun Jiao
  3. Xin Kang

Corresponding author

Correspondence toFuji Ren .

Editor information

Editors and Affiliations

  1. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
    Zhongzhi Shi
  2. Department of Computer Science, University of Surrey, Guildford, UK
    Yaochu Jin
  3. College of Artificial Intelligence, Xidian University, Xi’an, China
    Xiangrong Zhang

Rights and permissions

© 2022 IFIP International Federation for Information Processing

About this paper

Cite this paper

Ren, F., Jiao, Z., Kang, X. (2022). Improved Transformer-Based Implicit Latent GAN with Multi-headed Self-attention for Unconditional Text Generation. In: Shi, Z., Jin, Y., Zhang, X. (eds) Intelligence Science IV. ICIS 2022. IFIP Advances in Information and Communication Technology, vol 659. Springer, Cham. https://doi.org/10.1007/978-3-031-14903-0\_18

Download citation

Keywords

Publish with us