Central Algorithmic Techniques. Iterative Algorithms

Transcription

1 Central Algorithmic Techniques Iterative Algorithms

2 Code Representation of an Algorithm class InsertionSortAlgorithm extends SortAlgorithm { void sort(int a[]) throws Exception { for (int i = 1; i < a.length; i++) { int j = i; int B = a[i]; while ((j > 0) && (a[j-1] > B)) { a[j] = a[j-1]; j--; } a[j] = B; }} COSC 3101, PROF. J. ELDER 2 Pros and Cons?

3 Code Representation of an Algorithm Pros: Runs on computers Precise and succinct Cons: I am not a computer I need a higher level of intuition. Prone to bugs Language dependent We will focus on a more natural, intuitive and powerful method for developing, analyzing and proving correctness of iterative algorithms. This method will be based on loop invariants. COSC 3101, PROF. J. ELDER 3

4 Iterative Algorithms Take one step at a time towards the final destination loop (done) take step end loop COSC 3101, PROF. J. ELDER 4

5 Loop Invariants A good way to structure many programs: Store the key information you currently know in some data representation. In the main loop, take a step forward towards destination by making a simple change to this data. COSC 3101, PROF. J. ELDER 5

6 The Getting to School Problem COSC 3101, PROF. J. ELDER 6

7 Problem Specification Pre-condition: location of home and school Post-condition: Traveled from home to school COSC 3101, PROF. J. ELDER 7

8 General Principle Do not worry about the entire computation. Take one step at a time! COSC 3101, PROF. J. ELDER 8

9 A Measure of Progress 75 km to school 79 km to school COSC 3101, PROF. J. ELDER 9

10 Safe Locations Algorithm specifies from which locations it knows how to step. COSC 3101, PROF. J. ELDER 10

11 Loop Invariant The computation is presently in a safe location. May or may not be true. COSC 3101, PROF. J. ELDER 11

12 Defining Algorithm From every safe location, define one step towards school. COSC 3101, PROF. J. ELDER 12

13 Take a step What is required of this step? COSC 3101, PROF. J. ELDER 13

14 Maintain Loop Invariant If the computation is in a safe location, it does not step into an unsafe one. Can we be assured that the computation will always be in a safe location? No. What if it is not initially true? COSC 3101, PROF. J. ELDER 14

15 Establishing Loop Invariant From the Pre-Conditions on the input instance we must establish the loop invariant. COSC 3101, PROF. J. ELDER 15

16 Maintain Loop Invariant Suppose that We start in a safe location (pre-condition) If we are in a safe location, we always step to another safe location (loop invariant) Can we be assured that the computation will always be in a safe location? By what principle? COSC 3101, PROF. J. ELDER 16

17 Maintain Loop Invariant By Induction the computation will always be in a safe location. S(0) isi, ( ) isi, ( ) Si ( + 1) COSC 3101, PROF. J. ELDER 17

18 Ending The Algorithm Define Exit Condition Exit Termination: With sufficient progress, the exit condition will be met. 0 km Exit When we exit, we know exit condition is true loop invariant is true from these we must establish the post conditions. Exit COSC 3101, PROF. J. ELDER 18

19 Let s Recap COSC 3101, PROF. J. ELDER 19

20 Designing an Algorithm Define Problem Define Loop Invariants Define Measure of Progress 79 km to school Define Step Define Exit Condition Maintain Loop Inv Exit Exit Make Progress Initial Conditions Ending Exit 79 km 75 km km 0 km Exit Exit COSC 3101, PROF. J. ELDER 20

21 Simple Example Insertion Sort Algorithm

22 Code Representation of an Algorithm class InsertionSortAlgorithm extends SortAlgorithm { void sort(int a[]) throws Exception { for (int i = 1; i < a.length; i++) { int j = i; int B = a[i]; while ((j > 0) && (a[j-1] > B)) { a[j] = a[j-1]; j--; } a[j] = B; }} COSC 3101, PROF. J. ELDER 22

23 Higher Level Abstract View Representation of an Algorithm 5 km 9 km COSC 3101, PROF. J. ELDER 23

24 Problem Specification Pre-condition: The input is a list of n values with the same value possibly repeated. Post-condition: The output is a list consisting of the same n values in non-decreasing order ,23,25,30,31,52,62,79,88, COSC 3101, PROF. J. ELDER 24

25 Define Loop Invariant Some subset of the elements are sorted The remaining elements are off to the side ,23,25,30,31,52,62,79,88,98 23,31,52, COSC 3101, PROF. J. ELDER 25

26 Defining Measure of Progress 23,31,52, elements to school COSC 3101, PROF. J. ELDER 26

27 Define Step Select arbitrary element from side. Insert it where it belongs. 23,31,52, ,31,52,62, COSC 3101, PROF. J. ELDER 27

28 Making progress while maintaining the loop invariant 79 km 75 km Exit 23,31,52, elements to school 23,31,52,62, COSC 3101, PROF. J. ELDER 28 Sorted sublist 5 elements to school

29 Beginning & km 0 km Exit Exit Ending n elements to school 14,23,25,30,31,52,62,79,88,98 0 elements to school 14,23,25,30,31,52,62,79,88,98 COSC 3101, PROF. J. ELDER 29

30 Running Time Inserting an element into a list of size i takes θ(i) time. Total = n = θ(n 2 ) 23,31,52, COSC 3101, PROF. J. ELDER 30 23,31,52,62,

31 Ok: I know you knew Insertion Sort But hopefully you are beginning to appreciate the value of loop invariants for describing algorithms The loop invariant is an example of a more general thing called an assertion. COSC 3101, PROF. J. ELDER 31

32 Assertions in Algorithms

33 Purpose of Assertions Useful for thinking about algorithms developing describing proving correctness COSC 3101, PROF. J. ELDER 33

34 Definition of Assertions An assertion is a statement about the current state of the data structure that is either true or false. e.g., the amount in your bank account is not negative. COSC 3101, PROF. J. ELDER 34

35 Definition of Assertions It is made at some particular point during the execution of an algorithm. If it is false, then something has gone wrong in the logic of the algorithm. COSC 3101, PROF. J. ELDER 35

36 Definition of Assertions An assertion is not a task for the algorithm to perform. It is only a comment that is added for the benefit of the reader. COSC 3101, PROF. J. ELDER 36

37 Specifying a Computational Problem Example of Assertions Preconditions: Any assumptions that must be true about the input instance. Postconditions: The statement of what must be true when the algorithm/program returns. COSC 3101, PROF. J. ELDER 37

38 Definition of Correctness <PreCond> & <code> <PostCond> If the input meets the preconditions, then the output must meet the postconditions. If the input does not meet the preconditions, then nothing is required. COSC 3101, PROF. J. ELDER 38

39 <assertion 0 > if( <condition 1 > ) then else code <1,true> code <1,false> end if <assertion 1 > <assertion r-1 > if( <condition r > ) then else code <r,true> code <r,false> end if <assertion r > An Example: A Sequence of Assertions <assertion 0 > any <conditions> code COSC 3101, PROF. J. ELDER 39 Definition of Correctness <assertion r > How is this proved? Must check 2 r different settings of <conditions> paths through the code. Is there a faster way?

40 <assertion 0 > if( <condition 1 > ) then An Example: A Sequence of Assertions else code <1,true> code <1,false> end if <assertion 1 > <assertion r-1 > if( <condition r > ) then else code <r,true> code <r,false> end if Step 1 <assertion 0 > <condition 1 > code <1,true> <assertion 1 > <assertion 0 > <condition 1 > code <1,false> <assertion 1 > <assertion r > COSC 3101, PROF. J. ELDER 40

41 <assertion 0 > if( <condition 1 > ) then An Example: A Sequence of Assertions else code <1,true> code <1,false> end if <assertion 1 > <assertion r-1 > if( <condition r > ) then else code <r,true> code <r,false> end if Step 2 <assertion 1 > <condition 2 > code <2,true> <assertion 2 > <assertion 1 > <condition 2 > code <2,false> <assertion 2 > <assertion r > COSC 3101, PROF. J. ELDER 41

42 <assertion 0 > if( <condition 1 > ) then An Example: A Sequence of Assertions else code <1,true> code <1,false> end if <assertion 1 > <assertion r-1 > if( <condition r > ) then else code <r,true> code <r,false> end if Step r <assertion r-1 > <condition r > code <r,true> <assertion r > <assertion r-1 > <condition r > code <r,false> <assertion r > <assertion r > COSC 3101, PROF. J. ELDER 42

43 <assertion 0 > A Sequence of Assertions if( <condition 1 > ) then else code <1,true> code <1,false> end if <assertion 1 > <assertion r-1 > if( <condition r > ) then else code <r,true> code <r,false> end if We have reduced the number of cases r that must be considered from 2 to r. Step r <assertion r-1 > <condition r > code <r,true> <assertion r > <assertion r-1 > <condition r > code <r,false> <assertion r > <assertion r > COSC 3101, PROF. J. ELDER 43

44 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Another Example: A Loop Type of Algorithm: Iterative Type of Assertion: Loop Invariants COSC 3101, PROF. J. ELDER 44

45 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Iterative Algorithms Loop Invariants Definition of Correctness? COSC 3101, PROF. J. ELDER 45

46 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Iterative Algorithms Loop Invariants <precond> any <conditions> code Definition of Correctness How is this proved? <postcond> COSC 3101, PROF. J. ELDER 46

47 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> <precond> any <conditions> code The computation may go around the loop an arbitrary number of times. Is there a faster way? COSC 3101, PROF. J. ELDER 47 Iterative Algorithms Loop Invariants Definition of Correctness <postcond>

48 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Iterative Algorithms Loop Invariants <precond> codea Step 0 <loop-invariant COSC 3101, PROF. J. ELDER 48

49 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Iterative Algorithms Loop Invariants <loop-invariant> <exit Cond> codeb Step 1 <loop-invariant COSC 3101, PROF. J. ELDER 49

50 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Iterative Algorithms Loop Invariants <loop-invariant> <exit Cond> codeb Step 2 <loop-invariant COSC 3101, PROF. J. ELDER 50

51 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Iterative Algorithms Loop Invariants <loop-invariant> <exit Cond> codeb Step 3 <loop-invariant COSC 3101, PROF. J. ELDER 51

52 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> COSC 3101, PROF. J. ELDER 52 Iterative Algorithms Loop Invariants <loop-invariant> <exit Cond> codeb Step i <loop-invariant All these steps are the same and therefore only need be done once!

53 <precond> codea loop <loop-invariant> exit when <exit Cond> codeb endloop codec <postcond> Iterative Algorithms Loop Invariants <loop-invariant> <exit Cond> codec Last Step <postcond> COSC 3101, PROF. J. ELDER 53

54 Partial Correctness Establishing Loop Invariant <precond> codea <loop-invariant> Maintaining Loop Invariant Exit Clean up loose ends Exit <loop-invariant> <exit Cond> codeb <loop-invariant> <exit Cond> codec Proves that IF the program terminates then it works <PreCond> & <code> <PostCond> COSC 3101, PROF. J. ELDER 54 <loop-invariant> <postcond>

55 Algorithm Termination Measure of progress km 0 km Exit 79 km 75 km COSC 3101, PROF. J. ELDER 55

56 Algorithm Correctness Partial Correctness + Termination Correctness COSC 3101, PROF. J. ELDER 56

57 Designing Loop Invariants Coming up with the loop invariant is the hardest part of designing an algorithm. It requires practice, perseverance, and insight. Yet from it the rest of the algorithm follows easily COSC 3101, PROF. J. ELDER 57

58 Don t start coding You must design a working algorithm first. COSC 3101, PROF. J. ELDER 58

59 Exemplification: Try solving the problem on small input examples. COSC 3101, PROF. J. ELDER 59

60 Start with Small Steps What basic steps might you follow to make some kind of progress towards the answer? Describe or draw a picture of what the data structure might look like after a number of these steps. COSC 3101, PROF. J. ELDER 60

61 Picture from the Middle Leap into the middle of the algorithm. What would you like your data structure to look like when you are half done? COSC 3101, PROF. J. ELDER 61

62 Ask for 100% Pretend that a genie has granted your wish. You are now in the middle of your computation and your dream loop invariant is true. COSC 3101, PROF. J. ELDER 62

63 Ask for 100% Maintain the Loop Invariant: From here, are you able to take some computational steps that will make progress while maintaining the loop invariant? COSC 3101, PROF. J. ELDER 63

64 Ask for 100% If you can maintain the loop invariant, great. If not, Too Weak: If your loop invariant is too weak, then the genie has not provided you with everything you need to move on. Too Strong: If your loop invariant is too strong, then you will not be able to establish it initially or maintain it. COSC 3101, PROF. J. ELDER 64

65 Differentiating between Iterations x=x+2 Meaningful as code False as a mathematical statement x = x i = value at the beginning of the iteration x = x i+1 = new value after going around the loop one more time. x'' = x'+2 Meaningful as a mathematical statement COSC 3101, PROF. J. ELDER 65

66 Loop Invariants for Iterative Algorithms Three Search Examples

67 Define Problem: Binary Search PreConditions Key 25 Sorted List PostConditions Find key in list (if there) COSC 3101, PROF. J. ELDER 67

68 Define Loop Invariant Maintain a sublist. If the key is contained in the original list, then the key is contained in the sublist. key COSC 3101, PROF. J. ELDER 68

69 Define Step Make Progress Maintain Loop Invariant key COSC 3101, PROF. J. ELDER 69

70 Define Step Cut sublist in half. Determine which half the key would be in. Keep that half. key 25 mid If key mid, then key is in left half. COSC 3101, PROF. J. ELDER 70 If key > mid, then key is in right half.

71 Define Step It is faster not to check if the middle element is the key. Simply continue. key If key mid, then key is in left half. COSC 3101, PROF. J. ELDER 71 If key > mid, then key is in right half.

72 Make Progress Exit 79 km 75 km The size of the list becomes smaller km km COSC 3101, PROF. J. ELDER 72

73 Initial Conditions km key n km The sublist is the entire original list. If the key is contained in the original list, then the key is contained in the sublist. COSC 3101, PROF. J. ELDER 73

74 Ending Algorithm Exit key km If the key is contained in the original list, then the key is contained in the sublist. Sublist contains one element. Exit If the key is contained in the original list, then the key is at this location. COSC 3101, PROF. J. ELDER 74

75 If key not in original list If the key is contained in the original list, then the key is contained in the sublist. Loop invariant true, even if the key is not in the list. key If the key is contained in the original list, then the key is at this location. Conclusion still solves the problem. Simply check this one location for the key. COSC 3101, PROF. J. ELDER 75

76 Running Time The sublist is of size n, n / 2, n / 4, n / 8,,1 Each step θ(1) time. Total = θ(log n) key If key mid, then key is in left half. COSC 3101, PROF. J. ELDER 76 If key > mid, then key is in right half.

77 BinarySearch(A[1..n], key) <precondition>: A[1..n] is sorted in non-decreasing order <postcondition>: If key is in A[1..n], algorithm returns its location p = 1, q = n while q > p < loop-invariant>: If key is in A[1..n], then key is in A[p.. q] p + q mid = 2 if key A[ mid ] q = mid else p = mid + 1 end end if key = A[ p] return( p) else return("key not in list") end COSC 3101, PROF. J. ELDER 77

78 Algorithm Definition Completed Define Problem Define Loop Invariants Define Measure of Progress 79 km to school Define Step Define Exit Condition Maintain Loop Inv Exit Exit Make Progress Initial Conditions Ending Exit 79 km 75 km km 0 km Exit Exit COSC 3101, PROF. J. ELDER 78

79 Simple, right? Although the concept is simple, binary search is notoriously easy to get wrong. Why is this? COSC 3101, PROF. J. ELDER 79

80 The Devil in the Details The basic idea behind binary search is easy to grasp. It is then easy to write pseudocode that works for a typical case. Unfortunately, it is equally easy to write pseudocode that fails on the boundary conditions. COSC 3101, PROF. J. ELDER 80

81 The Devil in the Details if key A[ mid ] q = mid else p = mid + 1 end or if key < A[ mid ] q = mid else p = mid + 1 end What condition will break the loop invariant? COSC 3101, PROF. J. ELDER 81

82 The Devil in the Details key 36 mid Code: key A[mid] select right half Bug!! COSC 3101, PROF. J. ELDER 82

83 The Devil in the Details if key A[ mid ] q = mid else p = mid + 1 end if key < A[ mid ] q = mid 1 else p = mid end if key < A[ mid ] q = mid else p = mid + 1 end OK OK Not OK!! COSC 3101, PROF. J. ELDER 83

84 The Devil in the Details mid p+ q = 2 or mid p+ q = 2 key Shouldn t matter, right? Select mid p + q = 2 COSC 3101, PROF. J. ELDER 84

85 The Devil in the Details key 25 mid If key mid, then key is in left half. If key > mid, then key is in right half. COSC 3101, PROF. J. ELDER 85

86 The Devil in the Details key 25 mid If key mid, then key is in left half. COSC 3101, PROF. J. ELDER 86 If key > mid, then key is in right half.

87 The Devil in the Details Exit 79 km 75 km Another bug! key 25 mid No progress toward goal: Loops Forever! If key mid, then key is in left half. COSC 3101, PROF. J. ELDER 87 If key > mid, then key is in right half.

88 The Devil in the Details p + q mid = 2 if key A[ mid ] q = mid else p = mid + 1 end p + q mid = 2 if key < A[ mid ] q = mid 1 else p = mid end p + q mid = 2 if key A[ mid ] q = mid else p = mid + 1 end OK OK Not OK!! COSC 3101, PROF. J. ELDER 88

89 End of Lecture 4 March 16, 2009

90 Announcements Typo on Assignment 1: The last problem is worth 16 marks, not 30 marks. Check out the York programming contests. The first contest of the Winter term will take place this Friday March 20, 15:00-17:00 in CSEB COSC 3101, PROF. J. ELDER 90

91 How Many Possible Algorithms? p + q mid = 2 if key A[ mid ] q = mid else p = mid + 1 end p + q o r mid =? 2 or if key < A[ mid ]? or q = mid 1 else end p = mid COSC 3101, PROF. J. ELDER 91

92 Alternative Algorithm: Less Efficient but More Clear BinarySearch(A[1..n], key) <precondition>: A[1..n] is sorted in non-decreasing order <postcondition>: If key is in A[1..n], algorithm returns its location p = 1, q = n while q > p < loop-invariant>: If key is in A[1..n], then key is in A[p..q] p + q mid = 2 if key = A[ mid ] return(mid) elseif key < A[ mid ] q = mid 1 else p = mid + 1 end end if key = A[ p] return( p) else end return("key not in list" ) COSC 3101, PROF. J. ELDER 92 Still Θ(log n), but with slightly larger constant.

93 Moral Use the loop invariant method to think about algorithms. Be careful with your definitions. Be sure that the loop invariant is always maintained. Be sure progress is always made. Having checked the typical cases, pay particular attention to boundary conditions and the end game. Sometimes it is worth paying a little in efficiency for clear, correct code. COSC 3101, PROF. J. ELDER 93

94 Loop Invariants for Iterative Algorithms A Second Search Example: The Binary Search Tree

95 Define Problem: Binary Search Tree PreConditions Key 25 A binary search tree PostConditions Find key in BST (if there). COSC 3101, PROF. J. ELDER 95

96 Binary Search Tree All nodes in left subtree Any node All nodes in right subtree COSC 3101, PROF. J. ELDER

97 Maintain a sub-tree. Define Loop Invariant If the key is contained in the original tree, then the key is contained in the sub-tree. key COSC 3101, PROF. J. ELDER 97

98 Define Step Cut sub-tree in half. Determine which half the key would be in. Keep that half. key If key < root, then key is in left half. If key = root, then key is found If key > root, then key is in right half. COSC 3101, PROF. J. ELDER 98

99 Card Trick A volunteer, please. COSC 3101, PROF. J. ELDER 99

100 Loop Invariants for Iterative Algorithms A Third Search Example: A Card Trick

101 Pick a Card Done COSC 3101, PROF. J. ELDER 101

102 Loop Invariant: The selected card is one of these. COSC 3101, PROF. J. ELDER 102

103 Which column? left COSC 3101, PROF. J. ELDER 103

104 Loop Invariant: The selected card is one of these. COSC 3101, PROF. J. ELDER 104

105 Selected column is placed in the middle COSC 3101, PROF. J. ELDER 105

106 I will rearrange the cards COSC 3101, PROF. J. ELDER 106

107 Relax Loop Invariant: I will remember the same about each column. COSC 3101, PROF. J. ELDER 107

108 Which column? right COSC 3101, PROF. J. ELDER 108

109 Loop Invariant: The selected card is one of these. COSC 3101, PROF. J. ELDER 109

110 Selected column is placed in the middle COSC 3101, PROF. J. ELDER 110

111 I will rearrange the cards COSC 3101, PROF. J. ELDER 111

112 Which column? left COSC 3101, PROF. J. ELDER 112

113 Loop Invariant: The selected card is one of these. COSC 3101, PROF. J. ELDER 113

114 Selected column is placed in the middle COSC 3101, PROF. J. ELDER 114

115 Here is your card. Wow! COSC 3101, PROF. J. ELDER 115

116 Ternary Search Loop Invariant: selected card in central subset of cards i 1 Size of subset = n / 3 where n = total number of cards i = iteration index How many iterations are required to guarantee success? COSC 3101, PROF. J. ELDER 116

117 Loop Invariants for Iterative Algorithms A Fourth Example: Partitioning (Not a search problem: can be used for sorting, e.g., Quicksort)

118 The Partitioning Problem Input: x=52 Output: < Problem: Partition a list into a set of small values and a set of large values. COSC 3101, PROF. J. ELDER 118

119 Precise Specification Precondit ion: Ap [... r] is an arbitrary list of values. x= Ar [ ] is the pivot. p r Postcondition: A is rearranged such that A[ p... q 1] A[ q] = x < A[ q r] for some q. p q r COSC 3101, PROF. J. ELDER 119

120 Loop Invariant 3 subsets are maintained One containing values less than or equal to the pivot One containing values greater than the pivot One containing values yet to be processed COSC 3101, PROF. J. ELDER 120

121 Maintaining Loop Invariant Consider element at location j If greater than pivot, incorporate into > set by incrementing j. If less than or equal to pivot, incorporate into set by swapping with element at location i+1 and incrementing both i and j. Measure of progress: size of unprocessed set. COSC 3101, PROF. J. ELDER 121

122 Maintaining Loop Invariant COSC 3101, PROF. J. ELDER 122

123 Establishing Loop Invariant COSC 3101, PROF. J. ELDER 123

124 Establishing Postcondition = j on exit Exhaustive on exit COSC 3101, PROF. J. ELDER 124

125 Establishing Postcondition COSC 3101, PROF. J. ELDER 125

126 An Example COSC 3101, PROF. J. ELDER 126

127 Running Time Each iteration takes θ(1) time Total = θ(n) or COSC 3101, PROF. J. ELDER 127

128 Algorithm Definition Completed Define Problem Define Loop Invariants Define Measure of Progress 79 km to school Define Step Define Exit Condition Maintain Loop Inv Exit Exit Make Progress Initial Conditions Ending Exit 79 km 75 km km 0 km Exit Exit COSC 3101, PROF. J. ELDER 128

129 More Examples of Iterative Algorithms Using Constraints on Input to Achieve Linear- Time Sorting

130 Recall: InsertionSort nn ( + 1) n 2 Worst case (reverse order): tj = j : j = 1 T( n) θ( n ) j = 2 2 COSC 3101, PROF. J. ELDER 130

131 Recall: MergeSort COSC 3101, PROF. J. ELDER 131 Tn ( ) θ( nlog n)

132 Comparison Sorts InsertionSort and MergeSort are examples of (stable) Comparison Sort algorithms. QuickSort is another example we will study shortly. Comparison Sort algorithms sort the input by successive comparison of pairs of input elements. Comparison Sort algorithms are very general: they make no assumptions about the values of the input elements. COSC 3101, PROF. J. ELDER 132

133 Comparison Sorts 2 InsertionSort is θ( n ). MergeSort is θ( nlog n). Can we do better? COSC 3101, PROF. J. ELDER 133

134 Comparison Sort: Decision Trees Example: Sorting a 3-element array A[1..3] COSC 3101, PROF. J. ELDER 134

135 Comparison Sort Worst-case time is equal to the height of the binary decision tree. The height of the tree is the log of the number of leaves. The leaves of the tree represent all possible permutations of the input. How many are there? ( ) log n! Ω( nlog n) Thus MergeSort is asymptotically optimal. COSC 3101, PROF. J. ELDER 135

136 Linear Sorts? Comparison sorts are very general, but are Ω( nlog n) Faster sorting may be possible if we can constrain the nature of the input. COSC 3101, PROF. J. ELDER 136

137 Example 1. Counting Sort Counting Sort applies when the elements to be sorted come from a finite (and preferably small) set. For example, the elements to be sorted are integers in the range [0 k-1], for some fixed integer k. We can then create an array V[0 k-1] and use it to count the number of elements with each value [0 k-1]. Then each input element can be placed in exactly the right place in the output array in constant time. COSC 3101, PROF. J. ELDER 137

138 Counting Sort Input: Output: Input: N records with integer keys between [0 3]. Output: Stable sorted keys. Algorithm: Count frequency of each key value to determine transition locations Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 138

139 CountingSort Input: Output: Index: Stable sort: If two keys are the same, their order does not change. Thus the 4 th record in input with digit 1 must be the 4 th record in output with digit 1. It belongs at output index 8, because 8 records go before it ie, 5 records with a smaller digit & 3 records with the same digit Count These! COSC 3101, PROF. J. ELDER 139

140 Input: Output: CountingSort Index: Value v: # of records with digit v: N records. Time to count? Θ(N) COSC 3101, PROF. J. ELDER 140

141 Input: Output: CountingSort Index: Value v: # of records with digit v: # of records with digit < v: N records, k different values. Time to count? Θ(k) COSC 3101, PROF. J. ELDER 141

142 CountingSort Input: Output: Index: Value v: # of records with digit < v: = location of first record with digit v COSC 3101, PROF. J. ELDER 142

143 CountingSort Input: Output: Index: 0? Value v: Location of first record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 143

144 Loop Invariant The first i-1 keys have been placed in the correct locations in the output array The auxiliary data structure v indicates the location at which to place the i th key for each possible key value from [1..k-1]. COSC 3101, PROF. J. ELDER 144

145 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 145

146 CountingSort Input: Output: 0 1 Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 146

147 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 147

148 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 148

149 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 149

150 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 150

151 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 151

152 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 152

153 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 153

154 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 154

155 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 155

156 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 156

157 CountingSort Input: Output: Index: Value v: Location of next record with digit v Algorithm: Go through the records in order putting them where they go. COSC 3101, PROF. J. ELDER 157

158 CountingSort Input: Output: Index: Value v: Location of next record with digit v Time Total = Θ(N) Θ(N+k) COSC 3101, PROF. J. ELDER 158

159 Input: A of stack of N punch cards. Each card contains d digits. Each digit between [0 k-1] Output: Sorted cards. Digit Sort: Select one digit Example 2. RadixSort Separate cards into k piles based on selected digit (e.g., Counting Sort). Stable sort: If two cards are the same for that digit, their order does not change COSC 3101, PROF. J. ELDER 159

160 RadixSort Sort wrt which digit first? The most significant Sort wrt which digit Second? The next most significant All meaning in first sort lost. COSC 3101, PROF. J. ELDER 160

161 RadixSort Sort wrt which digit first? The least significant Sort wrt which digit Second? The next least significant COSC 3101, PROF. J. ELDER 161

162 RadixSort Sort wrt which digit first? The least significant Sort wrt which digit Second? The next least significant COSC 3101, PROF. J. ELDER 162 Is sorted wrt least sig. 2 digits.

163 RadixSort i+1 Is sorted wrt first i digits. Sort wrt i+1st digit COSC 3101, PROF. J. ELDER 163 Is sorted wrt first i+1 digits. These are in the correct order because sorted wrt high order digit

164 RadixSort i+1 Is sorted wrt first i digits. Sort wrt i+1st digit COSC 3101, PROF. J. ELDER 164 Is sorted wrt first i+1 digits. These are in the correct order because was sorted & stable sort left sorted

165 Loop Invariant The keys have been correctly stable-sorted with respect to the i-1 least-significant digits. COSC 3101, PROF. J. ELDER 165

166 Running Time Running time is Θ ( d( n + k)) Where d = # of digits in each number n = # of elements to be sorted k = # of possible values for each digit COSC 3101, PROF. J. ELDER 166

167 Example 3. Bucket Sort Applicable if input is constrained to finite interval, e.g., [0 1). If input is random and uniformly distributed, expected run time is Θ(n). COSC 3101, PROF. J. ELDER 167

168 Bucket Sort COSC 3101, PROF. J. ELDER 168

169 Loop Invariants Loop 1 The first i-1 keys have been correctly placed into buckets of width 1/n. Loop 2 The keys within each of the first i-1 buckets have been correctly stable-sorted. COSC 3101, PROF. J. ELDER 169

170 PseudoCode Expected Running Time Θ(1) Θ(1) Θ( n) Θ( n) n COSC 3101, PROF. J. ELDER 170

171 Examples of Iterative Algorithms Binary Search Partitioning Insertion Sort Counting Sort Radix Sort Bucket Sort Which can be made stable? Which sort in place? How about MergeSort? COSC 3101, PROF. J. ELDER 171

172 End of Iterative Algorithms