Fixed-point iteration (original) (raw)

From Wikipedia, the free encyclopedia

Root-finding algorithm

In numerical analysis, fixed-point iteration is a method of computing fixed points of a function.

More specifically, given a function f {\displaystyle f} $f$ defined on the real numbers with real values and given a point x 0 {\displaystyle x_{0}} $x_{0}$ in the domain of f {\displaystyle f} $f$ , the fixed-point iteration is x n + 1 = f ( x n ) , n = 0 , 1 , 2 , … {\displaystyle x_{n+1}=f(x_{n}),\,n=0,1,2,\dots } $x_{n+1}=f(x_{n}),\,n=0,1,2,\dots$ which gives rise to the sequence x 0 , x 1 , x 2 , … {\displaystyle x_{0},x_{1},x_{2},\dots } $x_{0},x_{1},x_{2},\dots$ of iterated function applications x 0 , f ( x 0 ) , f ( f ( x 0 ) ) , … {\displaystyle x_{0},f(x_{0}),f(f(x_{0})),\dots } $x_{0},f(x_{0}),f(f(x_{0})),\dots$ which is hoped to converge to a point x fix {\displaystyle x_{\text{fix}}} $x_{\text{fix}}$ . If f {\displaystyle f} $f$ is continuous, then one can prove that the obtained x fix {\displaystyle x_{\text{fix}}} $x_{\text{fix}}$ is a fixed point of f {\displaystyle f} $f$ , i.e., f ( x fix ) = x fix . {\displaystyle f(x_{\text{fix}})=x_{\text{fix}}.} $f(x_{\text{fix}})=x_{\text{fix}}.$

More generally, the function f {\displaystyle f} $f$ can be defined on any metric space with values in that same space.

The fixed-point iteration x n+1 = sin x n with initial value _x_0 = 2 converges to 0. This example does not satisfy the assumptions of the Banach fixed-point theorem and so its speed of convergence is very slow.

A first simple and useful example is the Babylonian method for computing the square root of a > 0, which consists in taking f ( x ) = 1 2 ( a x + x ) {\displaystyle f(x)={\frac {1}{2}}\left({\frac {a}{x}}+x\right)} $f(x)={\frac {1}{2}}\left({\frac {a}{x}}+x\right)$ , i.e. the mean value of x and a/x, to approach the limit x = a {\displaystyle x={\sqrt {a}}} $x={\sqrt {a}}$ (from whatever starting point x 0 ≫ 0 {\displaystyle x_{0}\gg 0} $x_{0}\gg 0$ ). This is a special case of Newton's method quoted below.
The fixed-point iteration x n + 1 = cos ⁡ x n {\displaystyle x_{n+1}=\cos x_{n}\,} $x_{n+1}=\cos x_{n}\,$ converges to the unique fixed point of the function f ( x ) = cos ⁡ x {\displaystyle f(x)=\cos x\,} $f(x)=\cos x\,$ for any starting point x 0 . {\displaystyle x_{0}.} $x_{0}.$ This example does satisfy (at the latest after the first iteration step) the assumptions of the Banach fixed-point theorem. Hence, the error after n steps satisfies | x n − x | ≤ q n 1 − q | x 1 − x 0 | = C q n {\displaystyle |x_{n}-x|\leq {q^{n} \over 1-q}|x_{1}-x_{0}|=Cq^{n}} $|x_{n}-x|\leq {q^{n} \over 1-q}|x_{1}-x_{0}|=Cq^{n}$ (where we can take q = 0.85 {\displaystyle q=0.85} $q=0.85$ , if we start from x 0 = 1 {\displaystyle x_{0}=1} $x_{0}=1$ .) When the error is less than a multiple of q n {\displaystyle q^{n}} $q^{n}$ for some constant q, we say that we have linear convergence. The Banach fixed-point theorem allows one to obtain fixed-point iterations with linear convergence.
The requirement that f is continuous is important, as the following example shows. The iteration x n + 1 = { x n 2 , x n ≠ 0 1 , x n = 0 {\displaystyle x_{n+1}={\begin{cases}{\frac {x_{n}}{2}},&x_{n}\neq 0\\1,&x_{n}=0\end{cases}}} $x_{n+1}={\begin{cases}{\frac {x_{n}}{2}},&x_{n}\neq 0\\1,&x_{n}=0\end{cases}}$ converges to 0 for all values of x 0 {\displaystyle x_{0}} $x_{0}$ . However, 0 is not a fixed point of the function f ( x ) = { x 2 , x ≠ 0 1 , x = 0 {\displaystyle f(x)={\begin{cases}{\frac {x}{2}},&x\neq 0\\1,&x=0\end{cases}}} $f(x)={\begin{cases}{\frac {x}{2}},&x\neq 0\\1,&x=0\end{cases}}$ as this function is not continuous at x = 0 {\displaystyle x=0} $x=0$ , and in fact has no fixed points.

Attracting fixed points

[edit]

The fixed point iteration x n+1 = cos x n with initial value _x_1 = −1.

An attracting fixed point of a function f is a fixed point _x_fix of f with a neighborhood U of "close enough" points around _x_fix such that for any value of x in U, the fixed-point iteration sequence x , f ( x ) , f ( f ( x ) ) , f ( f ( f ( x ) ) ) , … {\displaystyle x,\ f(x),\ f(f(x)),\ f(f(f(x))),\dots } $x,\ f(x),\ f(f(x)),\ f(f(f(x))),\dots$ is contained in U and converges to _x_fix. The basin of attraction of _x_fix is the largest such neighborhood U.[1]

The natural cosine function ("natural" means in radians, not degrees or other units) has exactly one fixed point, and that fixed point is attracting. In this case, "close enough" is not a stringent criterion at all—to demonstrate this, start with any real number and repeatedly press the cos key on a calculator (checking first that the calculator is in "radians" mode). It eventually converges to the Dottie number (about 0.739085133), which is a fixed point. That is where the graph of the cosine function intersects the line y = x {\displaystyle y=x} $y=x$ .[2]

Not all fixed points are attracting. For example, 0 is a fixed point of the function f(x) = 2_x_, but iteration of this function for any value other than zero rapidly diverges. We say that the fixed point of f ( x ) = 2 x {\displaystyle f(x)=2x} $f(x)=2x$ is repelling.

An attracting fixed point is said to be a stable fixed point if it is also Lyapunov stable.

A fixed point is said to be a neutrally stable fixed point if it is Lyapunov stable but not attracting. The center of a linear homogeneous differential equation of the second order is an example of a neutrally stable fixed point.

Multiple attracting points can be collected in an attracting fixed set.

Banach fixed-point theorem

[edit]

The Banach fixed-point theorem gives a sufficient condition for the existence of attracting fixed points. A contraction mapping function f {\displaystyle f} $f$ defined on a complete metric space has precisely one fixed point, and the fixed-point iteration is attracted towards that fixed point for any initial guess x 0 {\displaystyle x_{0}} $x_{0}$ in the domain of the function. Common special cases are that (1) f {\displaystyle f} $f$ is defined on the real line with real values and is Lipschitz continuous with Lipschitz constant L < 1 {\displaystyle L<1} $L<1$ , and (2) the function f is continuously differentiable in an open neighbourhood of a fixed point _x_fix, and | f ′ ( x fix ) | < 1 {\displaystyle |f'(x_{\text{fix}})|<1} $|f'(x_{\text{fix}})|<1$ .

Although there are other fixed-point theorems, this one in particular is very useful because not all fixed-points are attractive. When constructing a fixed-point iteration, it is very important to make sure it converges to the fixed point. We can usually use the Banach fixed-point theorem to show that the fixed point is attractive.

Attracting fixed points are a special case of a wider mathematical concept of attractors. Fixed-point iterations are a discrete dynamical system on one variable. Bifurcation theory studies dynamical systems and classifies various behaviors such as attracting fixed points, periodic orbits, or strange attractors. An example system is the logistic map.

In computational mathematics, an iterative method is a mathematical procedure that uses an initial value to generate a sequence of improving approximate solutions for a class of problems, in which the n-th approximation is derived from the previous ones. Convergent fixed-point iterations are mathematically rigorous formalizations of iterative methods.

Iterative method examples

[edit]

Newton's method is a root-finding algorithm for finding roots of a given differentiable function ⁠ f ( x ) {\displaystyle f(x)} $f(x)$ ⁠. The iteration is x n + 1 = x n − f ( x n ) f ′ ( x n ) . {\textstyle x_{n+1}=x_{n}-{\frac {f(x_{n})}{f'(x_{n})}}.} ${\textstyle x_{n+1}=x_{n}-{\frac {f(x_{n})}{f'(x_{n})}}.}$
If we write g ( x ) = x − f ( x ) f ′ ( x ) {\textstyle g(x)=x-{\frac {f(x)}{f'(x)}}} ${\textstyle g(x)=x-{\frac {f(x)}{f'(x)}}}$ , we may rewrite the Newton iteration as the fixed-point iteration x n + 1 = g ( x n ) {\textstyle x_{n+1}=g(x_{n})} ${\textstyle x_{n+1}=g(x_{n})}$ .
If this iteration converges to a fixed point x fix {\displaystyle x_{\text{fix}}} $x_{\text{fix}}$ of g, then x fix = g ( x fix ) = x fix − f ( x fix ) f ′ ( x fix ) {\textstyle x_{\text{fix}}=g(x_{\text{fix}})=x_{\text{fix}}-{\frac {f(x_{\text{fix}})}{f'(x_{\text{fix}})}}} ${\textstyle x_{\text{fix}}=g(x_{\text{fix}})=x_{\text{fix}}-{\frac {f(x_{\text{fix}})}{f'(x_{\text{fix}})}}}$ , so f ( x fix ) / f ′ ( x fix ) = 0 , {\textstyle f(x_{\text{fix}})/f'(x_{\text{fix}})=0,} ${\textstyle f(x_{\text{fix}})/f'(x_{\text{fix}})=0,}$
therefore f ( x fix ) = 0 {\displaystyle f(x_{\text{fix}})=0} $f(x_{\text{fix}})=0$ , that is, x fix {\displaystyle x_{\text{fix}}} $x_{\text{fix}}$ is a root of f {\displaystyle f} $f$ . Under the assumptions of the Banach fixed-point theorem, the Newton iteration, framed as a fixed-point method, demonstrates at least linear convergence. More detailed analysis shows quadratic convergence, i.e., | x n − x fix | < C q 2 n {\textstyle |x_{n}-x_{\text{fix}}|<Cq^{2^{n}}} ${\textstyle |x_{n}-x_{\text{fix}}|<Cq^{2^{n}}}$ , under certain circumstances.
Halley's method is similar to Newton's method when it works correctly, but its error is | x n − x fix | < C q 3 n {\displaystyle |x_{n}-x_{\text{fix}}|<Cq^{3^{n}}} $|x_{n}-x_{\text{fix}}|<Cq^{3^{n}}$ (cubic convergence). In general, it is possible to design methods that converge with speed C q k n {\displaystyle Cq^{k^{n}}} $Cq^{k^{n}}$ for any k ∈ N {\displaystyle k\in \mathbb {N} } $k\in \mathbb {N}$ . As a general rule, the higher the k, the less stable it is, and the more computationally expensive it gets. For these reasons, higher order methods are typically not used.
Runge–Kutta methods and numerical ordinary differential equation solvers in general can be viewed as fixed-point iterations. Indeed, the core idea when analyzing the A-stability of ODE solvers is to start with the special case y ′ = a y {\displaystyle y'=ay} $y'=ay$ , where a {\displaystyle a} $a$ is a complex number, and to check whether the ODE solver converges to the fixed point y fix = 0 {\displaystyle y_{\text{fix}}=0} $y_{\text{fix}}=0$ whenever the real part of a {\displaystyle a} $a$ is negative.[a]
The Picard–Lindelöf theorem, which shows that ordinary differential equations have solutions, is essentially an application of the Banach fixed-point theorem to a special sequence of functions which forms a fixed-point iteration, constructing the solution to the equation. Solving an ODE in this way is called Picard iteration, Picard's method, or the Picard iterative process.
The iteration capability in Excel can be used to find solutions to the Colebrook equation to an accuracy of 15 significant figures.[3][4]
Some of the "successive approximation" schemes used in dynamic programming to solve Bellman's functional equation are based on fixed-point iterations in the space of the return function.[5][6]
The cobweb model of price theory corresponds to the fixed-point iteration of the composition of the supply function and the demand function.[7]

Convergence acceleration

[edit]

The speed of convergence of the iteration sequence can be increased by using a convergence acceleration method such as Anderson acceleration and Aitken's delta-squared process. The application of Aitken's method to fixed-point iteration is known as Steffensen's method, and it can be shown that Steffensen's method yields a rate of convergence that is at least quadratic.

Sierpinski triangle created using IFS, selecting all members at each iteration

The term chaos game refers to a method of generating the fixed point of any iterated function system (IFS). Starting with any point _x_0, successive iterations are formed as x k+1 = f r(x k), where f r is a member of the given IFS randomly selected for each iteration. Hence the chaos game is a randomized fixed-point iteration. The chaos game allows plotting the general shape of a fractal such as the Sierpinski triangle by repeating the iterative process a large number of times. More mathematically, the iterations converge to the fixed point of the IFS. Whenever _x_0 belongs to the attractor of the IFS, all iterations x k stay inside the attractor and, with probability 1, form a dense set in the latter.

^ One may also consider certain iterations A-stable if the iterates stay bounded for a long time, which is beyond the scope of this article.
^ Rassias, Themistocles M.; Pardalos, Panos M. (17 September 2014). Mathematics Without Boundaries: Surveys in Pure Mathematics. Springer. ISBN 978-1-4939-1106-6.
^ Weisstein, Eric W. "Dottie Number". Wolfram MathWorld. Wolfram Research, Inc. Retrieved 23 July 2016.
^ M A Kumar (2010), Solve Implicit Equations (Colebrook) Within Worksheet, Createspace, ISBN 1-4528-1619-0
^ Brkic, Dejan (2017) Solution of the Implicit Colebrook Equation for Flow Friction Using Excel, Spreadsheets in Education (eJSiE): Vol. 10: Iss. 2, Article 2. Available at: https://sie.scholasticahq.com/article/4663-solution-of-the-implicit-colebrook-equation-for-flow-friction-using-excel
^ Bellman, R. (1957). Dynamic programming, Princeton University Press.
^ Sniedovich, M. (2010). Dynamic Programming: Foundations and Principles, Taylor & Francis.
^ Onozaki, Tamotsu (2018). "Chapter 2. One-Dimensional Nonlinear Cobweb Model". Nonlinearity, Bounded Rationality, and Heterogeneity: Some Aspects of Market Economies as Complex Systems. Springer. ISBN 978-4-431-54971-0.

Burden, Richard L.; Faires, J. Douglas (1985). "Fixed-Point Iteration". Numerical Analysis (Third ed.). PWS Publishers. ISBN 0-87150-857-5.
Hoffman, Joe D.; Frankel, Steven (2001). "Fixed-Point Iteration". Numerical Methods for Engineers and Scientists (Second ed.). New York: CRC Press. pp. 141–145. ISBN 0-8247-0443-6.
Judd, Kenneth L. (1998). "Fixed-Point Iteration". Numerical Methods in Economics. Cambridge: MIT Press. pp. 165–167. ISBN 0-262-10071-1.
Sternberg, Shlomo (2010). "Iteration and fixed points". Dynamical Systems (First ed.). Dover Publications. ISBN 978-0486477053.
Shashkin, Yuri A. (1991). "9. The Iteration Method". Fixed Points (First ed.). American Mathematical Society. ISBN 0-8218-9000-X.
Rosa, Alessandro (2021). "An episodic history of the staircased iteration diagram". Antiquitates Mathematicae. 15: 3–90. doi:10.14708/am.v15i1.7056. S2CID 247259939.
Fixed-point algorithms online
Fixed-point iteration online calculator (Mathematical Assistant on Web)