Bayesian Optimization over Mixed Type Inputs with Encoding Methods (original) (raw)

Abstract

Traditional Bayesian Optimization (BO) algorithms assume that the objective function is defined over numeric input space. To generalize BO for mixed numeric and categorical inputs, existing approaches mainly model or optimize them separately and thus cannot fully capture the relationship among different types of inputs. The complexity incurred by additional operations for the categorical inputs in these approaches can also reduce the efficiency of BO, especially when facing high-cardinality inputs. In this paper, we revisit the encoding approaches, which transfer categorical inputs to numerical ones to form a concise and easy-to-use BO framework. Specifically, we propose the target mean encoding BO (TmBO) and aggregate encoding BO (AggBO), where TmBO transfers each value of a categorical input based on the outputs corresponding to this value, and AggBO encodes multiple choices of a categorical input through several distinct ranks. Different from the prominent one-hot encoding, both approaches transfer each categorical input into exactly one numerical input and thus avoid severely increasing the dimension of the input space. We demonstrate that TmBO and AggBO are more efficient than existing approaches on several synthetic and real-world optimization tasks.

Similar content being viewed by others

References

  1. Abdelkhalik, O.: Autonomous planning of multigravity-assist trajectories with deep space maneuvers using a differential evolution approach. Int. J. Aerosp. Eng. 2013 (2013)
    Google Scholar
  2. Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems, vol. 24 (2011)
    Google Scholar
  3. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
    Article MATH Google Scholar
  4. Deshwal, A., Belakaria, S., Doppa, J.R.: Bayesian optimization over hybrid spaces. In: International Conference on Machine Learning, pp. 2632–2643. PMLR (2021)
    Google Scholar
  5. González, J., Dai, Z.: GPYOPT: a Bayesian optimization framework in Python. Accessed (2016)
    Google Scholar
  6. Häse, F., Aldeghi, M., Hickman, R.J., Roch, L.M., Aspuru-Guzik, A.: GRYFFIN: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge. Appl. Phys. Rev. 8(3), 031406 (2021)
    Article Google Scholar
  7. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40
    Chapter Google Scholar
  8. Lakshminarayanan, B., Roy, D.M., Teh, Y.W.: Mondrian forests for large-scale regression when uncertainty matters. In: Artificial Intelligence and Statistics, pp. 1478–1487. PMLR (2016)
    Google Scholar
  9. Larionov, M.: Sampling techniques in Bayesian target encoding. arXiv preprint arXiv:2006.01317 (2020)
  10. McCane, B., Albert, M.: Distance functions for categorical and mixed variables. Pattern Recogn. Lett. 29(7), 986–993 (2008)
    Article Google Scholar
  11. Micci-Barreca, D.: A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. ACM SIGKDD Explor. Newsl. 3(1), 27–32 (2001)
    Article Google Scholar
  12. Mockus, J., Tiesis, V., Zilinskas, A.: The application of Bayesian methods for seeking the extremum. Towards Glob. Optim. 2(117–129), 2 (1978)
    MATH Google Scholar
  13. Mougan, C., Masip, D., Nin, J., Pujol, O.: Quantile encoder: tackling high cardinality categorical features in regression problems. In: Torra, V., Narukawa, Y. (eds.) MDAI 2021. LNCS (LNAI), vol. 12898, pp. 168–180. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85529-1_14
    Chapter Google Scholar
  14. Nguyen, D., Gupta, S., Rana, S., Shilton, A., Venkatesh, S.: Bayesian optimization for categorical and category-specific continuous inputs. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 5256–5263 (2020)
    Google Scholar
  15. Pargent, F., Bischl, B., Thomas, J.: A benchmark experiment on how to encode categorical features in predictive modeling. Ph.D. thesis, Master Thesis in Statistics, Ludwig-Maximilians-Universität München ... (2019)
    Google Scholar
  16. Pargent, F., Pfisterer, F., Thomas, J., Bischl, B.: Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features. arXiv preprint arXiv:2104.00629 (2021)
  17. Ru, B., Alvi, A., Nguyen, V., Osborne, M.A., Roberts, S.: Bayesian optimisation over multiple continuous and categorical inputs. In: International Conference on Machine Learning, pp. 8276–8285. PMLR (2020)
    Google Scholar
  18. Sheather, S.J., Jones, M.C.: A reliable data-based bandwidth selection method for kernel density estimation. J. Roy. Stat. Soc.: Ser. B (Methodol.) 53(3), 683–690 (1991)
    MathSciNet MATH Google Scholar
  19. Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.: Gaussian process optimization in the bandit setting: no regret and experimental design. arXiv preprint arXiv:0912.3995 (2009)

Download references

Acknowledgement

This work is supported by the National Natural Science Foundation (NNSF) of China under Grant 72101106 and the Shenzhen Science and Technology Program under Grant No. RCBS20210609103119020.

Author information

Authors and Affiliations

  1. SUSTech Business School, Southern University of Science and Technology, Shenzhen, 518055, China
    Zhihao Liu, Weiming Ou & Songhao Wang

Authors

  1. Zhihao Liu
  2. Weiming Ou
  3. Songhao Wang

Corresponding author

Correspondence toSonghao Wang .

Editor information

Editors and Affiliations

  1. Kyoto University, Kyoto, Japan
    Hisashi Kashima
  2. IBM Research, Thomas J. Watson Research Center, Yorktown Heights, NY, USA
    Tsuyoshi Ide
  3. National Chiao Tung University, Hsinchu, Taiwan
    Wen-Chih Peng

Rights and permissions

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper

Liu, Z., Ou, W., Wang, S. (2023). Bayesian Optimization over Mixed Type Inputs with Encoding Methods. In: Kashima, H., Ide, T., Peng, WC. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2023. Lecture Notes in Computer Science(), vol 13936. Springer, Cham. https://doi.org/10.1007/978-3-031-33377-4\_16

Download citation

Publish with us