Differentiating a function you can only sample
Taylor gives you the formula and the error term in one line. Then floating point takes the error term back, and the result is the most important curve in the subject.
The definition of a derivative is a limit:
A machine cannot take a limit. It can only pick some small and evaluate the quotient. The entire question is: which , and how wrong are you?
Where the error term comes from
Taylor expand about :
Rearranging for :
So the forward difference is off by — an error proportional to . Halve the step, halve the error. That is what first order means, and the constant is not mysterious: it is literally half the second derivative.
Now do the same thing with a step in each direction and subtract:
Every even-powered term cancels by symmetry — including the term that was the leading error before. Divide through:
The central difference is second order, for the same two function evaluations. That is not a small improvement. At it is roughly ten thousand times more accurate, at identical cost.
Both schemes cost two evaluations of f. If you shrink h by a factor of 10, what happens to each error?
Option (d) is the interesting wrong answer, and it is worth sitting with, because everything above is exact mathematics and it genuinely does say the error goes to zero.
It just isn't what happens.
The curve that breaks the intuition
Watch what the error actually does as shrinks over fifteen orders of magnitude.
The left half is the theory working perfectly — straight lines whose slopes are the orders we just derived, 1 and 2 and 4.
Then every curve turns around.
The reason is the previous lesson. The numerator is a difference of nearly equal numbers, and as they become more nearly equal. Each is known to a relative precision of , so the absolute error in the numerator is about no matter how small gets. Dividing by then amplifies it:
Total error is the sum of the two effects, and they pull opposite ways:
Differentiate, set to zero, and you get the optimal step and the best error you can possibly achieve:
For the forward difference () that is and an error around — you can never get more than half your digits back, no matter how carefully you choose . For the central difference (), with error near .
Find the step size where the forward difference is at its most accurate. You are looking for the bottom of the valley, not the left edge of the plot.
h smaller eventually make a finite-difference derivative worse?Two error sources move in opposite directions as shrinks:
- Truncation error comes from cutting the Taylor series short. It falls.
- Roundoff error comes from subtracting nearly equal numbers and then dividing by a tiny quantity. It rises.
Their sum has a minimum at . Past that point you are paying more in lost significant digits than you gain in a better approximation to the limit.
The U-curve looks like a law of nature. It is not — it is a consequence of forming a difference. Remove the subtraction and the right-hand branch disappears.
Richardson extrapolation. If you know the error expands in known powers of , you can cancel the leading term algebraically. For the central difference,
Evaluate at and , then take the combination that eliminates :
Two second-order estimates, combined, give a fourth-order one. Iterating this is Romberg integration's differentiation cousin, and the same trick reappears in ODE solvers as extrapolation methods.
Complex-step differentiation. This one is close to magic. If is analytic and you can evaluate it in complex arithmetic, expand along the imaginary axis:
Take the imaginary part and divide by :
Second-order accurate — and crucially, there is no subtraction anywhere. Nothing cancels. So can be and the estimate is accurate to full machine precision. Turn the dashed curve on below and watch it simply refuse to bend.
The cost is real: must be analytic and implemented generically enough to accept complex inputs. When that holds — and in scientific code it often does — the technique gives derivatives at machine precision with one extra evaluation, and it was the standard trick for verifying hand-written Jacobians before automatic differentiation became routine.
For gradients of real programs, finite differences have largely been displaced by automatic differentiation, which computes derivatives by applying the chain rule to the computation graph itself. AD is exact to machine precision, costs a small constant factor over the function evaluation, and — decisively — computes a gradient in dimensions for roughly the cost of one function evaluation in reverse mode, where finite differences need .
Finite differences have not disappeared, though. They remain the tool of choice when:
- is a black box you cannot instrument (a legacy binary, an experiment, a vendor solver);
- you are verifying an analytic or AD gradient, where an independent method matters more than precision;
- you are discretising a PDE in space, where the finite-difference stencil is the numerical method rather than a way to estimate a derivative — and there is set by the grid, not by the U-curve.
That last case is the one that carries forward. The stencils here are the same objects that appear in finite-difference and finite-volume PDE solvers; the difference is that there, is a modelling choice, and the roundoff branch never comes into play.
What to carry forward
- Order of accuracy is a slope on a log-log plot, and you should always measure it rather than trust it. A method claiming fourth order that measures 2.1 has a bug.
- Symmetry buys accuracy for free. The central difference costs the same as the forward difference and is an order better. That idea reappears constantly.
- Smaller is not always better. Every method with a subtraction in it has a floor.
Next: the same machinery, but instead of estimating one derivative, we use it to follow a trajectory forward in time.