Error Propagation
While the errors in single floating-point numbers are very small, even simple calculations on them can contain pitfalls that increase the error in the result way beyond just having the individual errors “add up”.
In general:
- Multiplication and division are “safe” operations
- Addition and subtraction are dangerous:
- When numbers of different magnitudes are involved, digits of the smaller-magnitude number are lost. As an extreme example, if you have a single-precision floating point value of 100,000,000 and add 1 to it, the value will not change - even if you do it 100,000,000 times, because the result gets rounded back to 100,000,000 every single time.
- When numbers very close to each other are subtracted, the result’s less significant digits consist mostly of rounding errors - the more the closer the original numbers were. As an extreme example, in a single-precision calculation of the form
100,000,000 * (f1() - f2()) + 0,0001wheref1()andf2()are different functions that should return the same value, rounding errors will likely causef1() - f2()to be (incorrectly) non-zero. Magnified by multiplication with a large value, the final result could be many orders of magnitude bigger than the0,0001value it should have.
- Thus, the losses of precision can be inevitable and benign (when the lost digits also insignificant for the final result) or catastrophic (when the loss is magnified and distorts the result strongly).
- The more calculations are done (especially when they form an iterative algorithm) the more important it is to consider this kind of problem.
- A method of calculation can be stable (meaning that it tends to reduce rounding errors) or unstable (meaning that rounding errors are magnified). Very often, there are both stable and unstable solutions for a problem.
There is an entire sub-field of mathematics (in numerical analysis) devoted to studying the numerical stability of algorithms. For doing complex calculations involving floating-point numbers, it is absolutely necessary to have some understanding of this discipline.
The article What Every Computer Scientist Should Know About Floating-Point Arithmetic gives a detailed introduction, and served as an inspiration for creating this website, mainly due to being a bit too detailed and intimidating to programmers without a scientific background.
© Published at floating-point-gui.de under the Creative Commons Attribution License (BY)
