NewtonRaphson and Secant Methods
Newton's Method
The Background:
If we know how to describe mathematically the scalar values x we want,
we usually produce an equation describing them: "find x
such that (equation in x = 0.)" Think of the quadratic
equation, for instance: we need the x for which
a x^{2} +b x + c = 0.
Thus the goal is to find a value of
x
such that our function of interest,
f(x), is equal to zero.
That value of
x
is a root of the function. There are as many (real) roots as
places where the function crosses the xaxis. We assume the function
is
differentiable ("smooth") and that we can compute both it and its
derivative
Newton and secant are examples of the common engineering
trick of approximating an arbitrary function with a "firstorder"
function  in two dimensions, a straight line.
Later in life, you'll
expand functions into an infinite series (e.g. the Taylor series),
and pitch out all but a few larger, "leading" terms
to approximate the function close to a given point.
Nice article (with a movie), on
Newton's
method.
The Idea:
 Guess an
x_{0} close to the root of interest.
 Start Iteration: Approximate the function at that point by a straight line. The
obvious
choice is the line tangent to (in the direction of) the function's
graph at that point.
 Notice that the slope of the required tangent is the derivative of the
function, so the line we want has that slope and goes through the
point (x_{0}, f(x_{0})) .
 This tangent line goes through the xaxis at a point
x_{1}, which is easy to calculate and which we bet
is nearer to the root
than
x_{0}
is.
 Compute x_{1} and
f(x_{1}), and we're ready to go to Start Iteration
and repeat the process until for some
x_{i}, we find a
f(x_{i}) close enough to zero for our purposes.
Considered as an algorithm, this method
is clearly a while loop; it runs
until a smallerror condition is met.
The Math:
Say the tangent to the function at
x = x_{0}
intersects the xaxis at
x_{1}. The slope of that tangent is
Δy/ Δx = f'(x_{0}) (where f'
is the derivative of f), and
Δy = f(x_{0}).
Thus
f(x_{0})/(x_{0}  x_{1})
=f'(x_{0}), and so
x_{1} =
x_{0}  f(x_{0})/f'(x_{0})
.
We know everything on the RHS and the LHS is what we need to continue.
Repeat until done: Generally,
(Eq. 1) x_{i+1} =
x_{i}  f(x_{i})/f'(x_{i}).
Programming:
Super simple: about 10 lines of Matlab, calling (sub)functions
for f(x) and f'(x).
Extensions and Issues:

Many extensions, and various tweaks to the
method (use higherorder approximation functions, say).
 Extends to functions of several variables
(i.e. higherdimensional problems). In that case it uses the
Jacobian matrix you may see in vector calculus, and also the
generalized matrix inverse you'll definitely see in the DataFitting
segment later in 160.
Clearly there are potential problems.
 The process may actually
diverge,
not converge.
 Starting too far from the desired root may diverge or find some
other root.
To detect such problems and
gracefully abort, One could watch that the
error does not keep increasing for too long, or count iterations and
bail out after too many, etc.
See a more indepth treatment (like Wikipedia, say)
for more consumerprotection warnings.
Secant Method
The Background:
The
secant method has been around for thousands of years. The hook is it
does not use the derivative.
Same assumptions as for Newton, but
we use two initial points (ideally close to the root) and we
don't use the derivative, we approximate it with the secant line
to the curve (a cutting line: in the limit that the cut grazes the
function we have the tangent line).
Here's Wikipedia:
Secant
method.
The Idea:
 Pick two initial values of
x,
close to the desired root. Call them
x_{0}, x_{1}. Evaluate
y_{0} = f(x_{0}) and
y_{1} = f(x_{1}).
 As with the tangent line in Newton's method, produce the (secant)
line
through (x_{0}, y_{0}) and
(x_{1}, y_{1}), and compute where it
crosses the xaxis, and call that point
x_{2}. Get y_{2} =f(x_{2}).
 Bootstrap along: replace
(x_{0}, y_{0}),
(x_{1}, y_{1}) with
(x_{1}, y_{1}),
(x_{2}, y_{2}) and repeat.
 Keep this process up: derive
(x_{i}, y_{i}) from
(x_{i1}, y_{i1}) and
(x_{i2}, y_{i2}) until
y_{i} meets the error criterion.
This is the same while loop
control
structure as Newton, but needs a statement or two's worth of
bookkeeping since we need to
remember two previous x's, not one. ( Do Not save them all
in some vector, please! Always wasteful, sometimes dangerous, and no
easier to write).
The Math: Easy to formulate given we've done
Newton's method. Starting with Newton (Eq. 1), use
the "finitedifference" approximation:
f'(x_{i}) ≈ Δy/Δx =
(f(x_{i})  f(x_{i1}))
/(x_{i}  x_{i1}).
Thus for the secant method we need two initial x points,
which should be close to the desired root.
Generally,
(Eq. 2) x_{i+1} = x_{i}  f(x_{i})
[( x_{i}  x_{i1}) /
( f(x_{i})  f(x_{i1}))].
Extensions and Issues:

The most popular extension in one dimension is the method of false position,
(q.v.)
 There is also an extension to higherdimensional functions.
Same nonconvergence issues and answers as Newton, only risk is
greater with the approximation. But it sometimes works better!
The convergence rate is, stunningly enough,
the Golden Ratio,
which turns up all sorts of delightfully unexpected places, not just Greek
sculpture, Renaissance art, Fibonacci series, etc. Thus it is
about
1.6, slower than Newton but still better than linear.
Indeed, it may run
faster since it doesn't need to evaluate the derivative at every
step, or it may get lucky.
SimpleMinded Search
Step along until have two x
values, l and h , such that
f(l) < 0 and f(h)> 0.
Then search in the interval [l,h] for an
x such that f(x) is close enough to zero.
Binary search would be a good choice: as in "20 questions",
idea is to divide the unknown space into two equal parts at each
probe, so its size diminishes exponentially.
Find s, l : f(s) < 0, f(l) > 0
repeat
m = (s+l)/2;
if f(m) <= 0, s = m else l = m
until f(m) <= max_error_allowed;
Last Change: 9/23/2011: CB