THE BIG-O MEASURE OF COMPLEXITY:
We talk about the big-O of mathematical FUNCTIONS, usually
positive real-valued functions defined on the set of positive
integers. That is because we are usually interested in functions
that count the number of steps, or the amount of time, required
to carry out a given algorithm on a data set of a given size.
The size of the data set is always a positive integer, and the
number of steps or amount of time is always a positive real
number.
NOTATION:
The set of positive integers is usually denoted by the symbol N.
R denotes the set of real numbers -- the familiar set of numbers
corresponding to the points on a line. R+ denotes the set of
POSITIVE real numbers.
A positive real-valued function defined on N is a correspondence
or "rule" which ASSIGNS (pairs) A POSITIVE REAL NUMBER TO EACH
POSITIVE INTEGER.
Take as an example the case of a program that sorts lists. The
CORRESPONDENCE of a number to the average TIME required for the
program to sort a list containing that number of items is a
positive real-valued function on N. The various times are the
positive real numbers assigned by the rule.
We often denote functions using lower case letters, particularly
"f","g", "h", and "k". A familiar short-hand which states that
the function f is a positive real valued function defined on N is
"f: N --> R+".
We customarily denote the assigned value of a function by using
the notation f(s), where s is any symbol we are using to denote a
positive integer. For example, if f is the function described
above, then "f(3)" would stand for the average time required to
sort a list of 3 items, f(5) would stand for the average time
needed for sorting a list of 5 items. And, in general, if "s"
stands for a positive integer then "f(s)" stands for the average
time required to sort a list of s items.
DEFINITION OF BIG-O:
Suppose that f: N --> R+ and g: N --> R+.
If for SOME positive constants m and C,
(1) f(n) < [C * g(n)],
for EVERY positive integer n that is greater than m, then we say
that f is big-O of g (also written "f is O(g)").
For example, suppose that Frank's car is capable of great speed,
but takes quite a while to accelerate to its top speed. Suppose
that Gary's motorcycle is very quick to accelerate, but not
capable of the top speed of Frank's car. Gary may be able to
beat Frank in some short races, but there must be a number of
meters m such that Frank can beat Gary in any race of more than m
meters.
Let s be any positive integer. If f is the function that assigns
the time f(s) that Frank's car requires to race s meters, and if
g is the function that assigns the time g(s) that Gary's
motorcycle requires to race s meters, then according to our
definition of big-O, f is O(g).
Since Frank's elapsed time is less than Gary's elapsed time in
any race of more than m meters, f(n) < 1 * g(n) whenever n
> m. So the conditions given in line (1) hold true, with the
constant C having the value "1".
Now suppose that whenever Gary and Frank race, Frank is never
able to go 3 times as fast as Gary. Then Frank's times are more
than 1/3 of Gary's times, which is to say that g(s) < 3 * f(s)
for all s > 1. Thus g is O(f).
It may seem odd, but two positive real valued functions on N can
each be big-O of the other, and such pairs of functions are
considered roughly EQUIVALENT -- in mathematical parlance,
"ASYMPTOTICALLY PROPORTIONATE". f is big-O of g if and only if
f(n) < [C*g(n)] whenever n > m,
for some positive constants m and C. This is true if and only if
f(n)/g(n) < C whenever n > m,
and if and only if
g(n)/f(n) > 1/C when n > m.
On the other hand, if g is big-O of f, then for some positive K
and r,
g(a) < [K*f(a)] whenever a > r,
and this is the same as
g(a)/f(a) < K, and f(a)/g(a) > 1/K.
So
C > f(s)/g(s) > 1/K
whenever s is greater than both m and r. If f(s)/g(s) was a
constant, not depending at all on s, then we would say that f and
g are proportionate. The last inequality above shows that the
ratio of f(s) to g(s), while quite possibly not a constant, does
become "trapped" between C and 1/K for large enough values of s.
This phenomenon is called "asymptotic proportionality". We think
of the two functions as being "roughly proportionate".
Computer scientists classify algorithms by how they compare in
the "big-O" sense. Typically the functions f(n)=n, g(n)=n²,
k(n)=log(n), h(n)=n*log(n) are used as yardsticks to measure
algorithms.
If you know that a sorting algorithm requires big-O of n²
steps to sort n items , then generally speaking you can say that
it is inefficient. On the other hand, a sort that requires only
big-O of n*log(n) steps is generally considered quite efficient.
A more detailed analysis is needed to find out which method is
the better for any specific sorting task, but knowledge of the
big-O information is almost always the starting point for such an
analysis.