Week 04 Notes for CS 3100 -- Fall 2000

• Take Roll.
• Announcements
• There is a program due today. Questions? You may also look for me in the office area between 12:30 and about 3:00.
• We have a new programming assignment to start working on. Let's discuss the sorting algorithm and some initial program design considerations. Would any class member's like to suggest ideas for how the program should be structured -- say with modules or classes? Can we describe modules or classes that would be appropriate for this program?

• Take Roll.
• Announcement(s)
  • A copy of each of the course texts is now on reserve in the Library. The books are available for 2-hour checkout during the day, and for overnight checkout after 4:00 pm.
  • Students are reminded to please use the exact subject lines requested in the programming assignment documents.
• Discuss the efficiency of insertion sort
  • N-1 passes sort N items.
  • In the jth pass, item j+1 is inserted into a sorted sub-list of j elements.
  • There are j+1 positions within the sub-list in which item j+1 may belong.
  • Depending on which position within the list item j+1 takes, the number of comparisons required to find the position may be 1, 2, 3, ..., j, or j again if the new item belongs first in the new list.
  • We assume that each position is equally likely to be the right one for item j+1, so the average number of comparisons required to insert item j+1 is 1+2+3+ ... +j+j/(j+1) = (j/2) + 1 - (1/(j+1)).
  • The total average work for performing the insertion sort is the sum the average work done in each pass: it's the sum of terms of the form (j/2) + 1 - (1/(j+1)), for values of j from 1 to N-1.
  • Therefore the total average work is (N-1)N/4 + (N-1) - H, where H = (1/2)+(1/3)+...+(1/N).
  • We can get a very sharp estimate of the size of H using properties of logarithms, but a very rough estimate will serve our purpose here: certainly (1/2) <= H <= N-1.
  • Therefore the total average work (measured in number of comparisons) is between (N-1)N/4 and (N-1)N/4 + (N-1) - (1/2).
  • By far the most significant thing about the work function above is that the dominant term is N(N-1)/4, which is quadratic in N. This tells us that the number of comparisons required to perform an insertion sort of N elements is roughly proportionate to the square of N. The problem with this is that if we want to sort a list of moderately large size N, the square of N may be extremely large, and so the work required by insertion sort may be prohibitive.

• Take Roll.
• Announcements
• Discuss the efficiency of radix sort
  • Radix sort requires K passes to sort N K-digit items.
  • Radix sort is not based on comparisons between items. Instead, radix sort performs digit selection.
  • Each pass of the sort requires N digit selections. Therefore the entire sort requires N*K digit selections.
  • If we examine all the work required by radix sort, we find it is dominated by a term that appears to be linear in N: N*K.
  • Actually there is a "hidden" relationship between N and K. There is a limit to how many distinct items can be expressed with K digits. For example, there are only 10 distinct 1-digit numbers (including 0). There are only 100 2-digit numbers (again including 00). In general, the number of distinct K-digit numbers is the Kth power of 10. Thus we see that if we are sorting distinct numbers (no duplicates) then N <= 10^K (here "^" denotes exponentiation). This relation can be re-written as log(N) <= K, where "log" means the common logarithm (log to the base 10).
  • Therefore the amount of work required to sort N items with radix sort is roughly proportionate to N*K, which is more than N*log(N)
  • The function log(N) increases very "slowly" as N increases. Therefore when we consider sorting a large list of items, the use of radix sort can be feasible in situations where the use of insertion sort is not feasible.
• Compare the asymptotic efficiency of insertion sort with that of radix sort
  • We can get one measure of the relative efficiency of radix sort and insertion sort by looking at the ratio of the dominant terms of the work functions we computed earlier: N(N-1)/N*log(N) = (N-1)/log(N)
  • Using a little calculus (l'Hopital's rule) we find that the limit as N goes to infinity of the ratio is the same as the limit as N goes to infinity of this ratio: 1 / (1/N) = N.
  • The limit as N goes to infinity of "N" is of course infinity.
  • What are the implications of the findings above?
    • As the size of the list, N, increases the number of units of work required by insertion sort grows much greater than the number of units of work required by radix sort.
    • The ratio of the two quantities increases as N increases and there is no bound to how large that ratio grows.
    • No matter how much time the different individual units of work take (the comparisons, data swaps, digit selection, queue operations and so forth), there must be some value of N so large that the actual time taken by the insertion sort algorithm is more than the time taken by the radix sort algorithm. The point is that even if the individual operations required by insertion sort can be done much more quickly than the operations of radix sort, there will be so very many of these operations required in an insertion sort of N elements, and relatively so much fewer of the radix sort operations, that the total amount of time to sort N items with insertion sort will exceed the total time required to sort N items with radix sort.
    • Every list of size larger than the "magic" value of N referred to above will also require more work if done by insertion sort, and the difference in the amount of work will grow larger and larger without bound as the size of the list to sort increases.
    • We must be cautious to point out that this analysis does not indicate that insertion sort is always less efficient than radix sort. (That's not true!) It merely demonstrates that radix sort is asymptotically more efficient than insertion sort: if the list is large enough then radix sort will be faster.
    • This analysis also does not tell us how large "large enough" is above. That will depend on many details: how the algorithm is coded, how the data is represented, how efficient certain basic operations are on the computer used, and so forth.
• Summary: In the examples above we compute work functions for insertion sort and radix sort. We then compare the work functions in order to get information about the relative efficiency of the two algorithms. The example is important because it illustrates the kind of analysis that comoputer scientists frequently carry out when evaluating algorithms.