We’ve looked at Newton’s Method solely for finding roots, but there are other applications. One other significant use is locating local extreme points. This is the basis for many Optimization Algorithms. For those unfamiliar, Optimization problems consist of decision variables who take numeric values, and an objective function, which is to be maximized or minimized. They often also include constraints on the variables.

Newton’s method for finding local extrema uses the first three terms of the Taylor expansion, to produce a quadratic approximation of the function. Since Optimization problems are trivial in the single variable case, instead of using the first and second derivatives, the gradient and Hessian are used to produce the approximation. There is a single extreme point in the approximation since it is quadratic, and the location of the point is the initial point for the next iteration. Some divergence issues may arise if the full step is taken, so often only a fraction of the Newton step is added.

Similar to our interest in the secant method, there are methods that achieve a slower per iteration convergence rate for the benefit of less computation per iteration. These are called quasi-Newton methods, and attempt to circumvent the computationally expensive task of finding the Hessian matrix.

References:

http://en.wikipedia.org/wiki/Newton%27s_method_in_optimization

http://en.wikipedia.org/wiki/Quasi-Newton_method

http://www.it.lut.fi/kurssit/05-06/Ti5416300/lectures.html