The Complexity of Gradient Descent: CLS = PPAD ∩ PLS (original) (raw)

Article No.: 7, Pages 1 - 74

Published: 19 December 2022 Publication History

Abstract

We study search problems that can be solved by performing Gradient Descent on a bounded convex polytopal domain and show that this class is equal to the intersection of two well-known classes: PPAD and PLS. As our main underlying technical contribution, we show that computing a Karush-Kuhn-Tucker (KKT) point of a continuously differentiable function over the domain [0,1]2 is PPAD ∩ PLS-complete. This is the first non-artificial problem to be shown complete for this class. Our results also imply that the class CLS (Continuous Local Search) – which was defined by Daskalakis and Papadimitriou as a more “natural” counterpart to PPAD ∩ PLS and contains many interesting problems – is itself equal to PPAD ∩ PLS.

References

[1]

Ilan Adler, Christos Papadimitriou, and Aviad Rubinstein. 2014. On simplex pivoting rules and complexity theory. In Proceedings of the 17th Conference on Integer Programming and Combinatorial Optimization (IPCO’14). Springer, Bonn, 13–24. DOI:

[2]

Amir Ali Ahmadi and Jeffrey Zhang. 2022. Complexity aspects of local minima and related notions. Advances in Mathematics 397, Article 108119 (2022). DOI:

[3]

Amir Ali Ahmadi and Jeffrey Zhang. 2022. On the complexity of finding a local minimizer of a quadratic function over a polytope. Mathematical Programming (2022). DOI:

[4]

Eric Allender, Peter Bürgisser, Johan Kjeldgaard-Pedersen, and Peter Bro Miltersen. 2009. On the complexity of numerical analysis. SIAM Journal on Computing 38, 5 (2009), 1987–2006. DOI:

[5]

Yakov Babichenko and Aviad Rubinstein. 2021. Settling the complexity of Nash equilibrium in congestion games. In Proceedings of the 53rd ACM Symposium on Theory of Computing (STOC’21). ACM, 1426–1437. DOI:

[6]

Paul Beame, Stephen Cook, Jeff Edmonds, Russell Impagliazzo, and Toniann Pitassi. 1998. The relative complexity of NP search problems. Journal of Computer and System Sciences 57, 1 (1998), 3–19. DOI:

[7]

Dimitri P. Bertsekas. 1999. Nonlinear Programming. Athena Scientific.

[8]

Avrim Blum and Ronald L. Rivest. 1992. Training a 3-node neural network is NP-complete. Neural Networks 5, 1 (1992), 117–127. DOI:

[10]

Joshua Buresh-Oppenheim and Tsuyoshi Morioka. 2004. Relativized NP search problems and propositional proof systems. In Proceedings of the 19th IEEE Conference on Computational Complexity (CCC’04). IEEE, Amherst, 54–67. DOI:

[11]

Samuel R. Buss and Alan S. Johnson. 2012. Propositional proofs and reductions between NP search problems. Annals of Pure and Applied Logic 163, 9 (2012), 1163–1182. DOI:

[12]

Yair Carmon, John C. Duchi, Oliver Hinder, and Aaron Sidford. 2020. Lower bounds for finding stationary points I. Mathematical Programming 184 (2020), 71–120. DOI:

[13]

Augustin-Louis Cauchy. 1847. Méthode générale pour la résolution des systèmes d’équations simultanées. Comptes rendus de l’Académie des Sciences Paris 25 (1847), 536–538.

[15]

Xi Chen, Decheng Dai, Ye Du, and Shang-Hua Teng. 2009. Settling the complexity of Arrow-Debreu equilibria in markets with additively separable utilities. In Proceedings of the 50th IEEE Symposium on Foundations of Computer Science (FOCS’09). IEEE, Atlanta, 273–282. DOI:

[16]

Xi Chen and Xiaotie Deng. 2009. On the complexity of 2D discrete fixed point problem. Theoretical Computer Science 410, 44 (2009), 4448–4456. DOI:

[17]

Xi Chen, Xiaotie Deng, and Shang-Hua Teng. 2009. Settling the complexity of computing two-player Nash equilibria. Journal of the ACM 56, 3 (2009), 14:1–14:57. DOI:

[18]

Arka Rai Choudhuri, Pavel Hubáček, Chethan Kamath, Krzysztof Pietrzak, Alon Rosen, and Guy N. Rothblum. 2019. Finding a Nash equilibrium is no easier than breaking Fiat-Shamir. In Proceedings of the 51st ACM Symposium on Theory of Computing (STOC’19). ACM, Phoenix, 1103–1114. DOI:

[19]

Chuangyin Dang, Qi Qi, and Yinyu Ye. 2020. Computations and complexities of Tarski’s fixed points and supermodular games. (2020). arxiv:2005.09836

[20]

Constantinos Daskalakis, Paul W. Goldberg, and Christos H. Papadimitriou. 2009. The complexity of computing a Nash equilibrium. SIAM Journal on Computing 39, 1 (2009), 195–259. DOI:

[21]

Constantinos Daskalakis and Christos Papadimitriou. 2011. Continuous local search. In Proceedings of the 22nd ACM-SIAM Symposium on Discrete Algorithms (SODA’11). SIAM, San Francisco, 790–804. DOI:

[22]

Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. 2021. The complexity of constrained min-max optimization. In Proceedings of the 53rd ACM Symposium on Theory of Computing (STOC’21). ACM, 1466–1478. DOI:

[23]

Constantinos Daskalakis, Christos Tzamos, and Manolis Zampetakis. 2018. A converse to Banach’s fixed point theorem and its CLS-completeness. In Proceedings of the 50th ACM Symposium on Theory of Computing (STOC’18). ACM, Los Angeles, 44–50. DOI:

[24]

Leonardo Mendonça de Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems, 14th International Conference, TACAS. Springer, Budapest, 337–340. DOI:

[25]

Yann Disser and Martin Skutella. 2019. The simplex algorithm is NP-mighty. ACM Transactions on Algorithms 15, 1 (2019), 5:1–5:19. DOI:

[26]

Joydeep Dutta, Kalyanmoy Deb, Rupesh Tulshyan, and Ramnik Arora. 2013. Approximate KKT points and a proximity measure for termination. Journal of Global Optimization 56, 4 (2013), 1463–1499. DOI:

[27]

Kousha Etessami, Christos Papadimitriou, Aviad Rubinstein, and Mihalis Yannakakis. 2020. Tarski’s Theorem, supermodular games, and the complexity of equilibria. In Proceedings of the 11th Innovations in Theoretical Computer Science Conference (ITCS’20). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Seattle, 18:1–18:19. DOI:

[28]

Kousha Etessami and Mihalis Yannakakis. 2010. On the complexity of Nash equilibria and other fixed points. SIAM Journal on Computing 39, 6 (2010), 2531–2597. DOI:

[29]

Alex Fabrikant, Christos Papadimitriou, and Kunal Talwar. 2004. The complexity of pure Nash equilibria. In Proceedings of the 36th ACM Symposium on Theory of Computing (STOC’04). ACM, Chicago, 604–612. DOI:

[30]

John Fearnley, Spencer Gordon, Ruta Mehta, and Rahul Savani. 2017. CLS: New problems and completeness. (2017). arxiv:1702.06017

[31]

John Fearnley, Spencer Gordon, Ruta Mehta, and Rahul Savani. 2020. Unique end of potential line. J. Comput. System Sci. 114 (2020), 1–35. DOI:

[32]

John Fearnley, Dömötör Pálvölgyi, and Rahul Savani. 2022. A faster algorithm for finding Tarski fixed points. ACM Transactions on Algorithms 18, 3, Article 23 (2022). DOI:

[33]

John Fearnley and Rahul Savani. 2015. The complexity of the simplex method. In Proceedings of the 47th ACM Symposium on Theory of Computing (STOC). ACM, Portland, 201–208. DOI:

[34]

Paul W. Goldberg and Alexandros Hollender. 2021. The Hairy Ball problem is PPAD-complete. Journal of Computer and System Sciences 122 (2021), 34–62. DOI:

[35]

Mika Göös, Alexandros Hollender, Siddhartha Jain, Gilbert Maystre, William Pires, Robert Robere, and Ran Tao. 2022. Further collapses in TFNP. In Proceedings of the 37th Computational Complexity Conference (CCC’22). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Philadelphia, 33:1–33:15. DOI:

[36]

Pavel Hubáček and Eylon Yogev. 2020. Hardness of continuous local search: Query complexity and cryptographic lower bounds. SIAM Journal on Computing 49, 6 (2020), 1128–1172. DOI:

[37]

Takashi Ishizuka. 2021. The complexity of the parity argument with potential. Journal of Computer and System Sciences 120 (2021), 14–41. DOI:

[38]

Ruta Jawale, Yael Tauman Kalai, Dakshita Khurana, and Rachel Zhang. 2021. SNARGs for bounded depth computations and PPAD hardness from sub-exponential LWE. In Proceedings of the 53rd ACM Symposium on Theory of Computing (STOC’21). ACM, 708–721. DOI:

[39]

Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M. Kakade, and Michael I. Jordan. 2021. On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points. Journal of the ACM 68, 2 (2021), 11:1–11:29. DOI:

[40]

David S. Johnson, Christos H. Papadimitriou, and Mihalis Yannakakis. 1988. How easy is local search?Journal of Computer and System Sciences 37, 1 (1988), 79–100. DOI:

[41]

Shiva Kintali, Laura J. Poplawski, Rajmohan Rajaraman, Ravi Sundaram, and Shang-Hua Teng. 2013. Reducibility among fractional stability problems. SIAM Journal on Computing 42, 6 (2013), 2063–2113. DOI:

[42]

Donald E. Knuth. 1998. The Art of Computer Programming, Volume 3: Sorting and Searching. Addison-Wesley Professional.

[43]

M. K. Kozlov, S. P. Tarasov, and L. G. Khachiyan. 1980. The polynomial solvability of convex quadratic programming. USSR Computational Mathematics and Mathematical Physics 20, 5 (1980), 223–228. DOI:

[44]

Nimrod Megiddo and Christos H. Papadimitriou. 1991. On total functions, existence theorems and computational complexity. Theoretical Computer Science 81, 2 (1991), 317–324. DOI:

[45]

Ruta Mehta. 2018. Constant rank two-player games are PPAD-hard. SIAM Journal on Computing 47, 5 (2018), 1858–1887. DOI:

[46]

Frédéric Meunier, Wolfgang Mulzer, Pauline Sarrabezolles, and Yannik Stein. 2017. The rainbow at the end of the line–a PPAD formulation of the colorful Carathéodory theorem with applications. In Proceedings of the 28th ACM-SIAM Symposium on Discrete Algorithms (SODA’17). SIAM, Barcelona, 1342–1351. DOI:

[48]

Katta G. Murty and Santosh N. Kabadi. 1987. Some NP-complete problems in quadratic and nonlinear programming. Mathematical Programming 39, 2 (1987), 117–129. DOI:

[49]

Christos H. Papadimitriou. 1992. The complexity of the Lin-Kernighan heuristic for the traveling salesman problem. SIAM Journal on Computing 21, 3 (1992), 450–465. DOI:

[50]

Christos H. Papadimitriou. 1994. On the complexity of the parity argument and other inefficient proofs of existence. Journal of Computer and System Sciences 48, 3 (1994), 498–532. DOI:

[51]

Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. Annals of Mathematical Statistics 22, 3 (1951), 400–407. DOI:

[52]

William S. Russell. 1995. Polynomial interpolation schemes for internal derivative distributions on structured grids. Applied Numerical Mathematics 17, 2 (1995), 129–171. DOI:

[53]

Stephen A. Vavasis. 1993. Black-box complexity of local minimization. SIAM Journal on Optimization 3, 1 (1993), 60–80. DOI:

Information & Contributors

Information

Published In

cover image Journal of the ACM

Journal of the ACM Volume 70, Issue 1

February 2023

405 pages

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 December 2022

Online AM: 17 October 2022

Accepted: 07 October 2022

Revised: 20 April 2022

Received: 22 June 2021

Published in JACM Volume 70, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. TFNP
  2. computational complexity
  3. continuous optimization

Qualifiers

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

Reflects downloads up to 25 Dec 2024

Other Metrics

Citations

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Affiliations

John Fearnley

University of Liverpool, Liverpool, UK

Paul Goldberg

University of Oxford, Oxford, UK

Alexandros Hollender

University of Oxford, Oxford, UK

Rahul Savani

University of Liverpool, Liverpool, UK