diff --git a/lectures/1.org b/lectures/1.org index c4021b4..044b4f9 100644 --- a/lectures/1.org +++ b/lectures/1.org @@ -39,3 +39,28 @@ Since algorithms are finite, they have *bounded time* taken and *bounded space* + The Big Oh notation is used to define the upper bound of an algorithm. + Given a non negative funtion f(n) and other non negative funtion g(n), we say that $f(n) = O(g(n)$ if there exists a positive number $n_0$ and a positive constant $c$, such that \[ f(n) \le c.g(n) \ \ \forall n \ge n_0 \] + So if growth rate of g(n) is greater than or equal to growth rate of f(n), then $f(n) = O(g(n))$. + +** Omega Notation [ $\Omega$ ] ++ It is used to shown the lower bound of the algorithm. ++ For any positive integer $n_0$ and a positive constant $c$, we say that, $f(n) = \Omega (g(n))$ if \[ f(n) \ge c.g(n) \ \ \forall n \ge n_0 \] ++ So growth rate of $g(n)$ should be less than or equal to growth rate of $f(n)$ + +*Note* : If $f(n) = O(g(n))$ then $g(n) = \Omega (f(n))$ + +** Theta Notation [ $\theta$ ] ++ If is used to provide the asymptotic *equal bound*. ++ $f(n) = \theta (g(n))$ if there exists a positive integer $n_0$ and a positive constants $c_1$ and $c_2$ such that \[ c_1 . g(n) \le f(n) \le c_2 . g(n) \ \ \forall n \ge n_0 \] ++ So the growth rate of $f(n)$ and $g(n)$ should be equal. + +*Note* : So if $f(n) = O(g(n))$ and $f(n) = \Omega (g(n))$, then $f(n) = \theta (g(n))$ + +** Little-Oh Notation [o] ++ The little o notation defines the strict upper bound of an algorithm. ++ We say that $f(n) = o(g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) < c.g(n) \ \ \forall n \ge n_0 \] ++ Notice how condition is <, rather than $\le$ which is used in Big-Oh. So growth rate of $g(n)$ is strictly greater than that of $f(n)$. + +** Little-Omega Notation [ $\omega$ ] ++ The little omega notation defines the strict lower bound of an algorithm. ++ We say that $f(n) = \omega (g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) > c.g(n) \ \ \forall n \ge n_0 \] ++ Notice how condition is >, rather than $\ge$ which is used in Big-Omega. So growth rate of $g(n)$ is strictly less than that of $f(n)$. + diff --git a/lectures/2.org b/lectures/2.org index 58f0b8e..d7a4125 100644 --- a/lectures/2.org +++ b/lectures/2.org @@ -1,31 +1,4 @@ -* Asymptotic Notations - -** Omega Notation [ $\Omega$ ] -+ It is used to shown the lower bound of the algorithm. -+ For any positive integer $n_0$ and a positive constant $c$, we say that, $f(n) = \Omega (g(n))$ if \[ f(n) \ge c.g(n) \ \ \forall n \ge n_0 \] -+ So growth rate of $g(n)$ should be less than or equal to growth rate of $f(n)$ - -*Note* : If $f(n) = O(g(n))$ then $g(n) = \Omega (f(n))$ - -** Theta Notation [ $\theta$ ] -+ If is used to provide the asymptotic *equal bound*. -+ $f(n) = \theta (g(n))$ if there exists a positive integer $n_0$ and a positive constants $c_1$ and $c_2$ such that \[ c_1 . g(n) \le f(n) \le c_2 . g(n) \ \ \forall n \ge n_0 \] -+ So the growth rate of $f(n)$ and $g(n)$ should be equal. - -*Note* : So if $f(n) = O(g(n))$ and $f(n) = \Omega (g(n))$, then $f(n) = \theta (g(n))$ - -** Little-Oh Notation [o] -+ The little o notation defines the strict upper bound of an algorithm. -+ We say that $f(n) = o(g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) < c.g(n) \ \ \forall n \ge n_0 \] -+ Notice how condition is <, rather than $\le$ which is used in Big-Oh. So growth rate of $g(n)$ is strictly greater than that of $f(n)$. - -** Little-Omega Notation [ $\omega$ ] -+ The little omega notation defines the strict lower bound of an algorithm. -+ We say that $f(n) = \omega (g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) > c.g(n) \ \ \forall n \ge n_0 \] -+ Notice how condition is >, rather than $\ge$ which is used in Big-Omega. So growth rate of $g(n)$ is strictly less than that of $f(n)$. - * Comparing Growth rate of funtions - ** Applying limit To compare two funtions $f(n)$ and $g(n)$. We can use limit \[ \lim_{n\to\infty} \frac{f(n)}{g(n)} \] diff --git a/lectures/4.org b/lectures/4.org index 68e66f4..9db3cfa 100644 --- a/lectures/4.org +++ b/lectures/4.org @@ -54,7 +54,6 @@ Here, the recursive calls are func(n-1) and func(n-2), therefore time complexiti \[ T(n) = T(n-1) + T(n-2) + n \] \[ T(1) = T(0) = C\ \text{where C is constant time} \] - * Solving Recursive time complexities ** Iterative method + Take for example, @@ -165,9 +164,7 @@ Therefore, \theta (f(n)) = \theta (n^{log_ba}) So time complexity is, \[ T(n) = \theta ( n . (log(n))^{2}) \] - * Square root recurrence relations - ** Iterative method Example, \[ T(n) = T( \sqrt{n} ) + 1 \] @@ -247,3 +244,23 @@ Using master's theorem, \[ S(m) = \theta (m.1) \] Putting value of m, \[ T(n) = \theta (log(n)) \] + +* Extended Master's theorem for time complexity of recursive algorithms +** For (k = -1) +\[ T(n) = aT(n/b) + f(n).(log(n))^{-1} \] +\[ \text{Here, } f(n) \text{ is a polynomial function} \] +\[ a > 0\ and\ b > 1 \] + ++ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (n^{log_ba}) ++ If \theta (f(n)) > \theta ( n^{log_ba} ) then, T(n) = \theta (f(n)) ++ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (f(n).log(log(n))) + +** For (k < -1) +\[ T(n) = aT(n/b) + f(n).(log(n))^{k} \] +\[ \text{Here, } f(n) \text{ is a polynomial function} \] +\[ a > 0\ and\ b > 1\ and\ k < -1 \] + ++ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (n^{log_ba}) ++ If \theta (f(n)) > \theta ( n^{log_ba} ) then, T(n) = \theta (f(n)) ++ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (n^{log_ba}) + diff --git a/lectures/5.org b/lectures/5.org index 06ed92d..7b7fd8a 100644 --- a/lectures/5.org +++ b/lectures/5.org @@ -1,22 +1,3 @@ -* Extended Master's theorem for time complexity of recursive algorithms -** For (k = -1) -\[ T(n) = aT(n/b) + f(n).(log(n))^{-1} \] -\[ \text{Here, } f(n) \text{ is a polynomial function} \] -\[ a > 0\ and\ b > 1 \] - -+ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (n^{log_ba}) -+ If \theta (f(n)) > \theta ( n^{log_ba} ) then, T(n) = \theta (f(n)) -+ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (f(n).log(log(n))) - -** For (k < -1) -\[ T(n) = aT(n/b) + f(n).(log(n))^{k} \] -\[ \text{Here, } f(n) \text{ is a polynomial function} \] -\[ a > 0\ and\ b > 1\ and\ k < -1 \] - -+ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (n^{log_ba}) -+ If \theta (f(n)) > \theta ( n^{log_ba} ) then, T(n) = \theta (f(n)) -+ If \theta (f(n)) < \theta ( n^{log_ba} ) then, T(n) = \theta (n^{log_ba}) - * Tree method for time complexity of recursive algorithms Tree method is used when there are multiple recursive calls in our recurrance relation. Example, \[ T(n) = T(n/5) + T(4n/5) + f(n) \] diff --git a/lectures/6.org b/lectures/6.org index 9921388..35aa7f6 100644 --- a/lectures/6.org +++ b/lectures/6.org @@ -137,7 +137,6 @@ Recursive approach: /Another way to visualize binary search is using the binary tree./ * Max and Min element from array - ** Straightforward approach #+BEGIN_SRC C struc min_max {int min; int max;} @@ -257,3 +256,134 @@ def min_max(array): + Total number of comparisions = \[ \text{If n is odd}, \frac{3(n-1)}{2} \] \[ \text{If n is even}, \frac{3n}{2} - 2 \] + +* Square matrix multiplication + +Matrix multiplication algorithms taken from here: +[[https://www.cs.mcgill.ca/~pnguyen/251F09/matrix-mult.pdf]] + +** Straight forward method + +#+BEGIN_SRC C + /* This will calculate A X B and store it in C. */ + #define N 3 + + int main(){ + int A[N][N] = { + {1,2,3}, + {4,5,6}, + {7,8,9} }; + + int B[N][N] = { + {10,20,30}, + {40,50,60}, + {70,80,90} }; + + int C[N][N]; + + for(int i = 0; i < N; i++){ + for(int j = 0; j < N; j++){ + C[i][j] = 0; + for(int k = 0; k < N; k++){ + C[i][j] += A[i][k] * B[k][j]; + } + } + } + + return 0; + } +#+END_SRC + +Time complexity is $O(n^3)$ + +** Divide and conquer approach +The divide and conquer algorithm only works for a square matrix whose size is n X n, where n is a power of 2. The algorithm works as follows. + +#+BEGIN_SRC + MatrixMul(A, B, n): + If n == 2 { + return A X B + }else{ + Break A into four parts A_11, A_12, A_21, A_22, where A = [[ A_11, A_12], + [ A_21, A_22]] + + Break B into four parts B_11, B_12, B_21, B_22, where B = [[ B_11, B_12], + [ B_21, B_22]] + + C_11 = MatrixMul(A_11, B_11, n/2) + MatrixMul(A_12, B_21, n/2) + C_12 = MatrixMul(A_11, B_12, n/2) + MatrixMul(A_12, B_22, n/2) + C_21 = MatrixMul(A_21, B_11, n/2) + MatrixMul(A_22, B_21, n/2) + C_22 = MatrixMul(A_21, B_12, n/2) + MatrixMul(A_22, B_22, n/2) + + C = [[ C_11, C_12], + [ C_21, C_22]] + + return C + } +#+END_SRC + +The addition of matricies of size (n X n) takes time $\theta (n^2)$, therefore, for computation of C_11 will take time of $\theta \left( \left( \frac{n}{2} \right)^2 \right)$, which is equals to $\theta \left( \frac{n^2}{4} \right)$. Therefore, computation time of C_11, C_12, C_21 and C_22 combined will be $\theta \left( 4 \frac{n^2}{4} \right)$, which is equals to $\theta (n^2)$. +\\ +There are 8 recursive calls in this function with MatrixMul(n/2), therefore, time complexity will be +\[ T(n) = 8T(n/2) + \theta (n^2) \] +Using the *master's theorem* +\[ T(n) = \theta (n^{log_28}) \] +\[ T(n) = \theta (n^3) \] + +** Strassen's algorithm + +Another, more efficient divide and conquer algorithm for matrix multiplication. This algorithm also only works on square matrices with n being a power of 2. This algorithm is based on the observation that, for A X B = C. We can calculate C_11, C_12, C_21 and C_22 as, + +\[ \text{C_11 = P_5 + P_4 - P_2 + P_6} \] +\[ \text{C_12 = P_1 + P_2} \] +\[ \text{C_21 = P_3 + P_4} \] +\[ \text{C_22 = P_1 + P _5 - P_3 - P_7} \] +Where, +\[ \text{P_1 = A_11 X (B_12 - B_22)} \] +\[ \text{P_2 = (A_11 + A_12) X B_22} \] +\[ \text{P_3 = (A_21 + A_22) X B_11} \] +\[ \text{P_4 = A_22 X (B_21 - B_11)} \] +\[ \text{P_5 = (A_11 + A_22) X (B_11 + B_22)} \] +\[ \text{P_6 = (A_12 - A_22) X (B_21 + B_22)} \] +\[ \text{P_7 = (A_11 - A_21) X (B_11 + B_12)} \] +This reduces number of recursion calls from 8 to 7. + +#+BEGIN_SRC +Strassen(A, B, n): + If n == 2 { + return A X B + } + Else{ + Break A into four parts A_11, A_12, A_21, A_22, where A = [[ A_11, A_12], + [ A_21, A_22]] + + Break B into four parts B_11, B_12, B_21, B_22, where B = [[ B_11, B_12], + [ B_21, B_22]] + P_1 = Strassen(A_11, B_12 - B_22, n/2) + P_2 = Strassen(A_11 + A_12, B_22, n/2) + P_3 = Strassen(A_21 + A_22, B_11, n/2) + P_4 = Strassen(A_22, B_21 - B_11, n/2) + P_5 = Strassen(A_11 + A_22, B_11 + B_22, n/2) + P_6 = Strassen(A_12 - A_22, B_21 + B_22, n/2) + P_7 = Strassen(A_11 - A_21, B_11 + B_12, n/2) + C_11 = P_5 + P_4 - P_2 + P_6 + C_12 = P_1 + P_2 + C_21 = P_3 + P_4 + C_22 = P_1 + P_5 - P_3 - P_7 + C = [[ C_11, C_12], + [ C_21, C_22]] + return C + } +#+END_SRC + +This algorithm uses 18 matrix addition operations. So our computation time for that is $\theta \left(18\left( \frac{n}{2} \right)^2 \right)$ which is equal to $\theta (4.5 n^2)$ which is equal to $\theta (n^2)$. +\\ +There are 7 recursive calls in this function which are Strassen(n/2), therefore, time complexity is +\[ T(n) = 7T(n/2) + \theta (n^2) \] +Using the master's theorem +\[ T(n) = \theta (n^{log_27}) \] +\[ T(n) = \theta (n^{2.807}) \] + + ++ /*NOTE* : The divide and conquer approach and strassen's algorithm typically use n == 1 as their terminating condition since for multipliying 1 X 1 matrices, we only need to calculate product of the single element they contain, that product is thus the single element of our resultant 1 X 1 matrix./ + diff --git a/lectures/7.org b/lectures/7.org index 88bd4c1..9fd1874 100644 --- a/lectures/7.org +++ b/lectures/7.org @@ -1,133 +1,3 @@ -* Square matrix multiplication - -Matrix multiplication algorithms taken from here: -[[https://www.cs.mcgill.ca/~pnguyen/251F09/matrix-mult.pdf]] - -** Straight forward method - -#+BEGIN_SRC C - /* This will calculate A X B and store it in C. */ - #define N 3 - - int main(){ - int A[N][N] = { - {1,2,3}, - {4,5,6}, - {7,8,9} }; - - int B[N][N] = { - {10,20,30}, - {40,50,60}, - {70,80,90} }; - - int C[N][N]; - - for(int i = 0; i < N; i++){ - for(int j = 0; j < N; j++){ - C[i][j] = 0; - for(int k = 0; k < N; k++){ - C[i][j] += A[i][k] * B[k][j]; - } - } - } - - return 0; - } -#+END_SRC - -Time complexity is $O(n^3)$ - -** Divide and conquer approach -The divide and conquer algorithm only works for a square matrix whose size is n X n, where n is a power of 2. The algorithm works as follows. - -#+BEGIN_SRC - MatrixMul(A, B, n): - If n == 2 { - return A X B - }else{ - Break A into four parts A_11, A_12, A_21, A_22, where A = [[ A_11, A_12], - [ A_21, A_22]] - - Break B into four parts B_11, B_12, B_21, B_22, where B = [[ B_11, B_12], - [ B_21, B_22]] - - C_11 = MatrixMul(A_11, B_11, n/2) + MatrixMul(A_12, B_21, n/2) - C_12 = MatrixMul(A_11, B_12, n/2) + MatrixMul(A_12, B_22, n/2) - C_21 = MatrixMul(A_21, B_11, n/2) + MatrixMul(A_22, B_21, n/2) - C_22 = MatrixMul(A_21, B_12, n/2) + MatrixMul(A_22, B_22, n/2) - - C = [[ C_11, C_12], - [ C_21, C_22]] - - return C - } -#+END_SRC - -The addition of matricies of size (n X n) takes time $\theta (n^2)$, therefore, for computation of C_11 will take time of $\theta \left( \left( \frac{n}{2} \right)^2 \right)$, which is equals to $\theta \left( \frac{n^2}{4} \right)$. Therefore, computation time of C_11, C_12, C_21 and C_22 combined will be $\theta \left( 4 \frac{n^2}{4} \right)$, which is equals to $\theta (n^2)$. -\\ -There are 8 recursive calls in this function with MatrixMul(n/2), therefore, time complexity will be -\[ T(n) = 8T(n/2) + \theta (n^2) \] -Using the *master's theorem* -\[ T(n) = \theta (n^{log_28}) \] -\[ T(n) = \theta (n^3) \] - -** Strassen's algorithm - -Another, more efficient divide and conquer algorithm for matrix multiplication. This algorithm also only works on square matrices with n being a power of 2. This algorithm is based on the observation that, for A X B = C. We can calculate C_11, C_12, C_21 and C_22 as, - -\[ \text{C_11 = P_5 + P_4 - P_2 + P_6} \] -\[ \text{C_12 = P_1 + P_2} \] -\[ \text{C_21 = P_3 + P_4} \] -\[ \text{C_22 = P_1 + P _5 - P_3 - P_7} \] -Where, -\[ \text{P_1 = A_11 X (B_12 - B_22)} \] -\[ \text{P_2 = (A_11 + A_12) X B_22} \] -\[ \text{P_3 = (A_21 + A_22) X B_11} \] -\[ \text{P_4 = A_22 X (B_21 - B_11)} \] -\[ \text{P_5 = (A_11 + A_22) X (B_11 + B_22)} \] -\[ \text{P_6 = (A_12 - A_22) X (B_21 + B_22)} \] -\[ \text{P_7 = (A_11 - A_21) X (B_11 + B_12)} \] -This reduces number of recursion calls from 8 to 7. - -#+BEGIN_SRC -Strassen(A, B, n): - If n == 2 { - return A X B - } - Else{ - Break A into four parts A_11, A_12, A_21, A_22, where A = [[ A_11, A_12], - [ A_21, A_22]] - - Break B into four parts B_11, B_12, B_21, B_22, where B = [[ B_11, B_12], - [ B_21, B_22]] - P_1 = Strassen(A_11, B_12 - B_22, n/2) - P_2 = Strassen(A_11 + A_12, B_22, n/2) - P_3 = Strassen(A_21 + A_22, B_11, n/2) - P_4 = Strassen(A_22, B_21 - B_11, n/2) - P_5 = Strassen(A_11 + A_22, B_11 + B_22, n/2) - P_6 = Strassen(A_12 - A_22, B_21 + B_22, n/2) - P_7 = Strassen(A_11 - A_21, B_11 + B_12, n/2) - C_11 = P_5 + P_4 - P_2 + P_6 - C_12 = P_1 + P_2 - C_21 = P_3 + P_4 - C_22 = P_1 + P_5 - P_3 - P_7 - C = [[ C_11, C_12], - [ C_21, C_22]] - return C - } -#+END_SRC - -This algorithm uses 18 matrix addition operations. So our computation time for that is $\theta \left(18\left( \frac{n}{2} \right)^2 \right)$ which is equal to $\theta (4.5 n^2)$ which is equal to $\theta (n^2)$. -\\ -There are 7 recursive calls in this function which are Strassen(n/2), therefore, time complexity is -\[ T(n) = 7T(n/2) + \theta (n^2) \] -Using the master's theorem -\[ T(n) = \theta (n^{log_27}) \] -\[ T(n) = \theta (n^{2.807}) \] - - -+ /*NOTE* : The divide and conquer approach and strassen's algorithm typically use n == 1 as their terminating condition since for multipliying 1 X 1 matrices, we only need to calculate product of the single element they contain, that product is thus the single element of our resultant 1 X 1 matrix./ - * Sorting algorithms ** In place vs out place sorting algorithm diff --git a/lectures/8.org b/lectures/8.org index 5745186..e2d7229 100644 --- a/lectures/8.org +++ b/lectures/8.org @@ -133,6 +133,7 @@ For an array of size *n*, the number ofcomparisions done by this algorithm is al \[ T(n) = \theta (n) \] ** Time complexity of quicksort +In quick sort, we don't have a fixed recursive relation. The recursive relations differ for different cases. + *Best Case* : The partition algorithm always divides the array to two equal parts. In this case, the recursive relation becomes \[ T(n) = 2T(n/2) + \theta (n) \] Where, $\theta (n)$ is the time complexity for creating partition. @@ -147,4 +148,243 @@ For an array of size *n*, the number ofcomparisions done by this algorithm is al Using master's theorem \[ T(n) = \theta (n^2) \] -+ *Average Case* : ++ *Average Case* : The average case is closer to the best case in quick sort rather than to the worst case. +\\ +To get the average case, we will *consider a recursive function for number of comparisions* $C(n)$. +\\ +For the function $C(n)$, there are $n-1$ comparisions for the partition algorithm. +\\ +Now, suppose that the index of partition is *i*. +\\ +This will create two recursive comparisions $C(i)$ and $C(n-i-1)$. +\\ +*i* can be any number between *0* and *n-1*, with each case being equally probable. So the average number of comparisions for $C(n)$ will be +\[ \frac{1}{n} \sum_{i=0}^{n-1} \left( C(i) + C(n-i-1) \right) \] +Therefore, total number of comparisions for input size *n* will be, +\[ C(n) = \left( n-1 \right) + \frac{1}{n} \sum_{i=0}^{n-1} \left( C(i) + C(n-i-1) \right) \] +Solving the above recurrance relation will give us, +\[ C(n) \approx 2\ n\ ln(n) \] +\[ C(n) \approx 1.39\ n\ log_2(n) \] +Therefore, the time complexity in average case becomes, +\[ T(n) = \theta (n\ log_2(n)) \] + +** Number of comparisions +The number of comparisions in quick sort for, ++ Worst Case : \[ \text{Number of comparisions} = \frac{n(n-1)}{2} \] + +* Merging two sorted arrays (2-Way Merge) +Suppose we have two arrays that are already sorted. The first array has *n* elements and the second array has *m* elements. +\\ +The way to merge them is to compare the elements in a sequence between the two arrays. We first add a pointer to start of both arrays. The element pointed by the pointers are compared and the smaller one is added to our new array. Then we move pointer on that array forward. These comparisions are repeated until we reach the end of one of the array. At this point, we can simply append all the elements of the remaining array. + +#+BEGIN_SRC C + int *merge(int a[], int n, int b[], int m){ + int *c = malloc((m+n) * sizeof(int)); + int i = 0; int j = 0; + int k = 0; + + while (i != n && j != m) { + if ( a[i] > b[j] ) { c[k++] = b[j++]; } else { c[k++] = a[i++]; }; + } + + while (i != n) { + c[k++] = a[i++]; + } + + while (j != m) { + c[k++] = b[j++]; + } + + return c; + } +#+END_SRC + ++ The maximum number of comparisions to merge the arrays is (m + n - 1). ++ The minimum number of comparisions to merge the arrays is either *m* or *n*. Depending of which one is smaller. + +* Merging k sorted arrays (k-way merge) +k-way merge algorithms take k different sorted arrays and merge them into a single single array. The algorithm is same as that in two way merge except we need to get the smallest element from the pointer on k array's and then move it's corresponding pointer. + +* Merge sort +Merge sort is a pure divide and conquer algorithm. In this sorting algorithm, we merge the sorted sub-arrays till we get a final sorted array.\\ +The algorithm will work as follows : +1. Divide the array of n elements into *n* subarrays, each having one element. +2. Repeatdly merge the subarrays to form merged subarrays of larger sizes until there is one list remaining. + +For divide and conquer steps: ++ *Divide* : Divide the array from the middle into two equal sizes. ++ *Conquer* : Call merge sort recursively on the two subarrays ++ *Combine* : Merge the sorted array + +The algorithm works as follows (this isn't real c code) +#+BEGIN_SRC C + // A function that will merge two sorted arrays + int[] merge(int first[], int second[]); + + int[] merge_sort(int array[], int left, int right){ + if(left < right){ + int mid = (left + right) / 2; + int sorted_first[] = merge_sort(array[], left, mid); + int sorted_second[] = merge_sort(array[], mid + 1, right); + + return merge(sorted_first, sorted_second); + } + } +#+END_SRC + +This algorithm is often used in languages which have great support for linked lists, for example lisp and haskell. For more traditional c-like languages, often quicksort is easier to implement. +\\ +An implementation in C language is as follows. + +#+BEGIN_SRC C + // buffer is memory of size equal to or bigger than size of array + // buffer is used when merging the arrays + void merge_sort(int array[], int left, int right, int buffer[]){ + if(left < right){ + // Divide part + int mid = ( left + right ) / 2; + + // Conquer part + merge_sort(array,left, mid, buffer); + merge_sort(array, mid + 1, right, buffer); + + // Combine part : Merges the two sorted parts + int i = left; int j = mid + 1; int k = 0; + while( i != (mid+1) && j != (right+1) ){ + if(array[i] < array[j]) { buffer[k++] = array[i++]; } else { buffer[k++] = array[j++]; } + } + + while(i != (mid+1)) + buffer[k++] = array[i++]; + + while(j != (right+1)) + buffer[k++] = array[j++]; + + for(int x = left; x <= right; x++) + array[x] = buffer[x - left]; + } + } +#+END_SRC + +** Time complexity +Unlike quick sort, *the recurrance relation is same for merge sort in all cases.* +\\ +Since divide part divides array into two equal sizes, the input size is halfed (i.e, *T(n/2)* ). +\\ +In conquer part, there are two calls so *2.T(n/2)* is added to time complexity. +\\ +The cost for merging two arrays of size n/2 each is either *n-1* of *n/2*. That is to say that time complexity to merge two arrays of size n/2 each is always $\theta (n)$. Thus, the final recurrance relation is +\[ T(n) = 2.T(n/2) + \theta (n) \] +Using the master's theorem. +\[ T(n) = \theta (n.log_2n) \] + +** Space complexity +As we can see in the C code, the space complexity is $\theta (n)$ + +* Stable and unstable sorting algorithms +We call sorting algorithms unstable or stable on the basis of whether they change order of equal values. ++ *Stable sorting algorithm* : a sorting algorithm that preserves the order of the elements with equal values. ++ *Unstable sorting algorithm* : a sorting algorithm that does not preserve the order of the elements with equal values. + \\ +This is of importance when we store data in pairs of keys and values and then sort data using the keys. So we may want to preserve the order in which the entries where added. +\\ +Example, suppose we add (key, value) pairs as: +#+BEGIN_SRC + (2, v1), (1, v2), (3, v3), (1, v1), (2, v4), (3, v2) +#+END_SRC + +Now, if we sort using the keys a sorting algorithm which is stabe will preserve the order of elements with equal keys. So output is always +#+BEGIN_SRC + (1, v2), (1, v1), (2,v1), (2, v4), (3, v3), (3, v2) +#+END_SRC +i.e, the *order of keys with same values is preserved*. +\\ +Whereas an unstable sorting algorithm will sort without preserving the order of key values. + +* Non-comparitive sorting algorithms +Sorting algorithms which do not use comparisions to sort elements are called non-comparitive sorting algorithms. These tend to be faster than comparitive sorting algorithms. + +** Counting sort ++ Counting sort *only works on integer arrays* ++ Couting sort only works if *all elements of array are non-negative*, i.e, elements are only allowed to be in range [0,k] . + +#+BEGIN_SRC c + //* The input array is sorted and result is stored in output array *// + //* max is the largest element of the array *// + void counting_sort(int input[], int max ,int output[]){ + // count array should have a size greater than or equal to (max + 1) + int count[max + 1]; + // initialize count array to zero, can also use memset + for(int i = 0; i < max+1; i++) count[i] = 0; + + // i from 0 to len(array) - 1 + // this loop stores number of elements equal to i in count array + for(int i = 0; i < len(array); i++) + count[input[i]] = count[input[i]] + 1; + + // i from 1 to max + // this loop stores number of elements less that or equal to i in count array + for(int i = 1; i <= max; i++) + count[i] = count[i] + count[i - 1]; + + // i from len(array) - 1 to 0 + for(int i = len(array) - 1; i >= 0; i--){ + count[input[i]] = count[input[i]] - 1; + output[count[input[i]]] = input[i]; + } + } +#+END_SRC + ++ *Time complexity* : Since there are only simple loops and arithmetic operations, we can get time complexity by considering the number of times loops are executed. + + \[ \text{Number of times loops are executed} = n + (max - 1) + n \] + \[ \text{Where, } n = len(array) \text{ i.e, the input size} \] + + Therefore, + \[ \text{Number of times loops are executed} = 2n + max - 1 \] + \[ \text{Time complexity} = \theta (n + max) \] + +** Radix sort +In radix sort, we sort using the digits, from least significant digit (lsd) to most significant digit (msd). In other words, we sort digits from right to left. The algorithm used to sort digits *should be a stable sorting algorithm*. + +[[./imgs/radix-sort.png]] + +For the following example, we will use the bubble sort since it is the easiest to implement. But, for best performance, *radix sort is paired with counting sort*. + +#+BEGIN_SRC c + // d = 0, will return digit at unit's place + // d = 1, will return digit at ten's place + // and so on. + int get_digit(int n, int d){ + assert(d >= 0); + int place = (int) pow(10, d); + int digit = (n / place) % 10; + return digit; + } + + // bubble sort the array for only digits of the given place + // d = 0, unit's place + // d = 1, ten's place + // and so on. + void bubble_sort_digit(int array[], int d){ + for(int i = len(array); i >= 1; i--){ + for(int j = 0; j < i; j++){ + if(get_digit(array[j], d) > get_digit(array[j + 1], d)) + array[j], array[j + 1] = array[j + 1], array[j]; + } + } + } + + void radix_sort(int array[], int no_of_digits){ + for(int i = 0; i < no_of_digits ; i++){ + bubble_sort_digit(array, i ); + } + } +#+END_SRC + ++ *Time complexity* : \[ \text{Time Complexity} = \theta (d.(n + max)) \] + Where, *d = number of digits in max elemet*, and + \\ + radix sort is paired with counting sort. + +** Bucket sort diff --git a/lectures/imgs/radix-sort.png b/lectures/imgs/radix-sort.png new file mode 100644 index 0000000..68405d9 Binary files /dev/null and b/lectures/imgs/radix-sort.png differ diff --git a/main.html b/main.html index 2068476..c7c4696 100644 --- a/main.html +++ b/main.html @@ -3,7 +3,7 @@ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> - + Algorithms @@ -44,163 +44,176 @@

Table of Contents

-
-

1. Lecture 1

+
+

1. Lecture 1

-
-

1.1. Data structure and Algorithm

+
+

1.1. Data structure and Algorithm

  • A data structure is a particular way of storing and organizing data. The purpose is to effectively access and modify data effictively.
    • @@ -227,8 +240,8 @@ During programming we use data structures and algorithms that work on that data.
-
-

1.2. Characteristics of Algorithms

+
+

1.2. Characteristics of Algorithms

An algorithm has follwing characteristics. @@ -244,8 +257,8 @@ An algorithm has follwing characteristics.

-
-

1.3. Behaviour of algorithm

+
+

1.3. Behaviour of algorithm

The behaviour of an algorithm is the analysis of the algorithm on basis of Time and Space. @@ -262,8 +275,8 @@ The preference is traditionally/usually given to better time complexity. But we

-
-

1.3.1. Best, Worst and Average Cases

+
+

1.3.1. Best, Worst and Average Cases

The input size tells us the size of the input given to algorithm. Based on the size of input, the time/storage usage of the algorithm changes. Example, an array with larger input size (more elements) will taken more time to sort. @@ -276,8 +289,8 @@ The input size tells us the size of the input given to algorithm. Based on the s

-
-

1.3.2. Bounds of algorithm

+
+

1.3.2. Bounds of algorithm

Since algorithms are finite, they have bounded time taken and bounded space taken. Bounded is short for boundries, so they have a minimum and maximum time/space taken. These bounds are upper bound and lower bound. @@ -290,12 +303,12 @@ Since algorithms are finite, they have bounded time taken and bounded

-
-

1.4. Asymptotic Notations

+
+

1.4. Asymptotic Notations

-
-

1.4.1. Big-Oh Notation [O]

+
+

1.4.1. Big-Oh Notation [O]

  • The Big Oh notation is used to define the upper bound of an algorithm.
    • @@ -304,19 +317,10 @@ Since algorithms are finite, they have bounded time taken and bounded
-
-
-
-

2. Lecture 2

-
-
-
-

2.1. Asymptotic Notations

-
-
-
-

2.1.1. Omega Notation [ \(\Omega\) ]

-
+ +
+

1.4.2. Omega Notation [ \(\Omega\) ]

+
  • It is used to shown the lower bound of the algorithm.
  • For any positive integer \(n_0\) and a positive constant \(c\), we say that, \(f(n) = \Omega (g(n))\) if \[ f(n) \ge c.g(n) \ \ \forall n \ge n_0 \]
    • @@ -329,9 +333,9 @@ Since algorithms are finite, they have bounded time taken and bounded
-
-

2.1.2. Theta Notation [ \(\theta\) ]

-
+
+

1.4.3. Theta Notation [ \(\theta\) ]

+
  • If is used to provide the asymptotic equal bound.
  • \(f(n) = \theta (g(n))\) if there exists a positive integer \(n_0\) and a positive constants \(c_1\) and \(c_2\) such that \[ c_1 . g(n) \le f(n) \le c_2 . g(n) \ \ \forall n \ge n_0 \]
    • @@ -344,9 +348,9 @@ Since algorithms are finite, they have bounded time taken and bounded
-
-

2.1.3. Little-Oh Notation [o]

-
+
+

1.4.4. Little-Oh Notation [o]

+
  • The little o notation defines the strict upper bound of an algorithm.
  • We say that \(f(n) = o(g(n))\) if there exists positive integer \(n_0\) and positive constant \(c\) such that, \[ f(n) < c.g(n) \ \ \forall n \ge n_0 \]
    • @@ -355,9 +359,9 @@ Since algorithms are finite, they have bounded time taken and bounded
-
-

2.1.4. Little-Omega Notation [ \(\omega\) ]

-
+
+

1.4.5. Little-Omega Notation [ \(\omega\) ]

+
  • The little omega notation defines the strict lower bound of an algorithm.
  • We say that \(f(n) = \omega (g(n))\) if there exists positive integer \(n_0\) and positive constant \(c\) such that, \[ f(n) > c.g(n) \ \ \forall n \ge n_0 \]
    • @@ -366,14 +370,18 @@ Since algorithms are finite, they have bounded time taken and bounded
- -
-

2.2. Comparing Growth rate of funtions

-
-
-

2.2.1. Applying limit

-
+
+

2. Lecture 2

+
+
+
+

2.1. Comparing Growth rate of funtions

+
+
+
+

2.1.1. Applying limit

+

To compare two funtions \(f(n)\) and \(g(n)\). We can use limit \[ \lim_{n\to\infty} \frac{f(n)}{g(n)} \] @@ -389,9 +397,9 @@ To compare two funtions \(f(n)\) and \(g(n)\). We can use limit

-
-

2.2.2. Using logarithm

-
+
+

2.1.2. Using logarithm

+

Using logarithm can be useful to compare exponential functions. When comaparing functions \(f(n)\) and \(g(n)\),

@@ -404,9 +412,9 @@ Using logarithm can be useful to compare exponential functions. When comaparing
-
-

2.2.3. Common funtions

-
+
+

2.1.3. Common funtions

+

Commonly, growth rate in increasing order is \[ c < c.log(log(n)) < c.log(n) < c.n < n.log(n) < c.n^2 < c.n^3 < c.n^4 ... \] @@ -417,13 +425,13 @@ Where \(c\) is any constant.

-
-

2.3. Properties of Asymptotic Notations

-
+
+

2.2. Properties of Asymptotic Notations

+
-
-

2.3.1. Big-Oh

-
+
+

2.2.1. Big-Oh

+
  • Product : \[ Given\ f_1 = O(g_1)\ \ and\ f_2 = O(g_2) \implies f_1 f_2 = O(g_1 g_2) \] \[ Also\ f.O(g) = O(f g) \]
    • @@ -434,11 +442,11 @@ Where \(c\) is any constant.
-
-

2.3.2. Properties

-
+
+

2.2.2. Properties

+
-
+

asymptotic-notations-properties.png

@@ -455,12 +463,12 @@ Where \(c\) is any constant.
-
-

3. Lecture 3

+
+

3. Lecture 3

-
-

3.1. Calculating time complexity of algorithm

+
+

3.1. Calculating time complexity of algorithm

We will look at three types of situations @@ -472,8 +480,8 @@ We will look at three types of situations

-
-

3.1.1. Sequential instructions

+
+

3.1.1. Sequential instructions

A sequential set of instructions are instructions in a sequence without iterations and recursions. It is a simple block of instructions with no branches. A sequential set of instructions has time complexity of O(1), i.e., it has constant time complexity. @@ -481,8 +489,8 @@ A sequential set of instructions are instructions in a sequence without iteratio

-
-

3.1.2. Iterative instructions

+
+

3.1.2. Iterative instructions

A set of instructions in a loop. Iterative instructions can have different complexities based on how many iterations occurs depending on input size. @@ -686,8 +694,8 @@ Time complexity = \(O(n.log(n))\)

-
-

3.1.3. An example for time complexities of nested loops

+
+

3.1.3. An example for time complexities of nested loops

Suppose a loop, @@ -781,20 +789,20 @@ Putting value \(k = log(n)\)

-
-

4. Lecture 4

+
+

4. Lecture 4

-
-

4.1. Time complexity of recursive instructions

+
+

4.1. Time complexity of recursive instructions

To get time complexity of recursive functions/calls, we first also show time complexity as recursive manner.

-
-

4.1.1. Time complexity in recursive form

+
+

4.1.1. Time complexity in recursive form

We first have to create a way to describe time complexity of recursive functions in form of an equation as, @@ -870,13 +878,12 @@ Here, the recursive calls are func(n-1) and func(n-2), therefore time complexiti

- -
-

4.2. Solving Recursive time complexities

+
+

4.2. Solving Recursive time complexities

-
-

4.2.1. Iterative method

+
+

4.2.1. Iterative method

  • Take for example,
    • @@ -957,8 +964,8 @@ Time complexity is

-
-

4.2.2. Master Theorem for Subtract recurrences

+
+

4.2.2. Master Theorem for Subtract recurrences

For recurrence relation of type @@ -988,8 +995,8 @@ Since a > 1, \(T(n) = O(n^2 . 3^n)\)

-
-

4.2.3. Master Theorem for divide and conquer recurrences

+
+

4.2.3. Master Theorem for divide and conquer recurrences

\[ T(n) = aT(n/b) + f(n).(log(n))^k \] @@ -1044,13 +1051,12 @@ So time complexity is,

- -
-

4.3. Square root recurrence relations

+
+

4.3. Square root recurrence relations

-
-

4.3.1. Iterative method

+
+

4.3.1. Iterative method

Example, @@ -1092,8 +1098,8 @@ Time complexity is,

-
-

4.3.2. Master Theorem for square root recurrence relations

+
+

4.3.2. Master Theorem for square root recurrence relations

For recurrence relations with square root, we need to first convert the recurrance relation to the form with which we use master theorem. Example, @@ -1148,18 +1154,14 @@ Putting value of m,

+ +
+

4.4. Extended Master's theorem for time complexity of recursive algorithms

+
-
-

5. Lecture 5

-
-
-
-

5.1. Extended Master's theorem for time complexity of recursive algorithms

-
-
-
-

5.1.1. For (k = -1)

-
+
+

4.4.1. For (k = -1)

+

\[ T(n) = aT(n/b) + f(n).(log(n))^{-1} \] \[ \text{Here, } f(n) \text{ is a polynomial function} \] @@ -1174,9 +1176,9 @@ Putting value of m,

-
-

5.1.2. For (k < -1)

-
+
+

4.4.2. For (k < -1)

+

\[ T(n) = aT(n/b) + f(n).(log(n))^{k} \] \[ \text{Here, } f(n) \text{ is a polynomial function} \] @@ -1191,10 +1193,14 @@ Putting value of m,

- -
-

5.2. Tree method for time complexity of recursive algorithms

-
+
+
+

5. Lecture 5

+
+
+
+

5.1. Tree method for time complexity of recursive algorithms

+

Tree method is used when there are multiple recursive calls in our recurrance relation. Example, \[ T(n) = T(n/5) + T(4n/5) + f(n) \] @@ -1322,9 +1328,9 @@ Of the two possible time complexities, we consider the one with higher growth ra

-
-

5.2.1. Avoiding tree method

-
+
+

5.1.1. Avoiding tree method

+

The tree method as mentioned is mainly used when we have multiple recursive calls with different factors. But when using the big-oh notation (O). We can avoid tree method in favour of the master's theorem by converting recursive call with smaller factor to larger. This works since big-oh calculates worst case. Let's take our previous example \[ T(n) = T(n/5) + T(4n/5) + f(n) \] @@ -1342,9 +1348,9 @@ Now, our recurrance relation is in a form where we can apply the mater's theorem

-
-

5.3. Space complexity

-
+
+

5.2. Space complexity

+

The amount of memory used by the algorithm to execute and produce the result for a given input size is space complexity. Similar to time complexity, when comparing two algorithms space complexity is usually represented as the growth rate of memory used with respect to input size. The space complexity includes

@@ -1358,9 +1364,9 @@ The amount of memory used by the algorithm to execute and produce the result for

-
-

5.3.1. Auxiliary space complexity

-
+
+

5.2.1. Auxiliary space complexity

+

The space complexity when we disregard the input space is the auxiliary space complexity, so we basically treat algorithm as if it's input space is zero. Auxiliary space complexity is more useful when comparing algorithms because the algorithms which are working towards same result will have the same input space, Example, the sorting algorithms will all have the input space of the list, so it is not a metric we can use to compare algorithms. So from here, when we calculate space complexity, we are trying to calculate auxiliary space complexity and sometimes just refer to it as space complexity.

@@ -1368,9 +1374,9 @@ The space complexity when we disregard the input space is the auxiliary space co
-
-

5.4. Calculating auxiliary space complexity

-
+
+

5.3. Calculating auxiliary space complexity

+

There are two parameters that affect space complexity,

@@ -1380,9 +1386,9 @@ There are two parameters that affect space complexity,
-
-

5.4.1. Data Space used

-
+
+

5.3.1. Data Space used

+

The data space used by the algorithm depends on what data structures it uses to solve the problem. Example,

@@ -1436,9 +1442,9 @@ Here, we create a matrix of size n*n, so the increase in allocated space
-
-

5.4.2. Code Execution space in recursive algorithm

-
+
+

5.3.2. Code Execution space in recursive algorithm

+

When we use recursion, the function calls are stored in the stack. This means that code execution space will increase. A single function call has fixed (constant) space it takes in the memory. So to get space complexity, we need to know how many function calls occur in the longest branch of the function call tree.

@@ -1528,12 +1534,12 @@ Number of levels is \(log_2n\). Therefore, space complexity is \(\theta (log_
-
-

6. Lecture 6

+
+

6. Lecture 6

-
-

6.1. Divide and Conquer algorithms

+
+

6.1. Divide and Conquer algorithms

Divide and conquer is a problem solving strategy. In divide and conquer algorithms, we solve problem recursively applying three steps : @@ -1556,12 +1562,12 @@ Divide and conquer is a problem solving strategy. In divide and conquer algorith

-
-

6.2. Searching for element in array

+
+

6.2. Searching for element in array

-
-

6.2.1. Straight forward approach for searching (Linear Search)

+
+

6.2.1. Straight forward approach for searching (Linear Search)

int linear_search(int *array, int n, int x){
@@ -1684,8 +1690,8 @@ Recursive approach
-
-

6.2.2. Divide and conquer approach (Binary search)

+
+

6.2.2. Divide and conquer approach (Binary search)

The binary search algorithm works on an array which is sorted. In this algorithm we: @@ -1709,7 +1715,7 @@ Suppose binarySearch(array, left, right, key), left and right are indicies of le -

+

binary-search.jpg

@@ -1780,12 +1786,12 @@ Recursive approach:
-
-

6.3. Max and Min element from array

+
+

6.3. Max and Min element from array

-
-

6.3.1. Straightforward approach

+
+

6.3.1. Straightforward approach

struc min_max {int min; int max;}
@@ -1813,8 +1819,8 @@ Recursive approach:
-
-

6.3.2. Divide and conquer approach

+
+

6.3.2. Divide and conquer approach

Suppose the function is MinMax(array, left, right) which will return a tuple (min, max). We will divide the array in the middle, mid = (left + right) / 2. The left array will be array[left:mid] and right aray will be array[mid+1:right] @@ -1893,8 +1899,8 @@ If n is not a power of 2, we will round the number of comparision up.

-
-

6.3.3. Efficient single loop approach (Increment by 2)

+
+

6.3.3. Efficient single loop approach (Increment by 2)

In this algorithm we will compare pairs of numbers from the array. It works on the idea that the larger number of the two in pair can be the maximum number and smaller one can be the minimum one. So after comparing the pair, we can simply test from maximum from the bigger of two an minimum from smaller of two. This brings number of comparisions to check two numbers in array from 4 (when we increment by 1) to 3 (when we increment by 2). @@ -1940,23 +1946,19 @@ In this algorithm we will compare pairs of numbers from the array. It works on t

-
-
-

7. Lecture 7

-
-
-
-

7.1. Square matrix multiplication

-
+ +
+

6.4. Square matrix multiplication

+

Matrix multiplication algorithms taken from here: https://www.cs.mcgill.ca/~pnguyen/251F09/matrix-mult.pdf

-
-

7.1.1. Straight forward method

-
+
+

6.4.1. Straight forward method

+
/* This will calculate A X B and store it in C. */
 #define N 3
@@ -1994,9 +1996,9 @@ Time complexity is \(O(n^3)\)
-
-

7.1.2. Divide and conquer approach

-
+
+

6.4.2. Divide and conquer approach

+

The divide and conquer algorithm only works for a square matrix whose size is n X n, where n is a power of 2. The algorithm works as follows.

@@ -2036,9 +2038,9 @@ Using the master's theorem
-
-

7.1.3. Strassen's algorithm

-
+
+

6.4.3. Strassen's algorithm

+

Another, more efficient divide and conquer algorithm for matrix multiplication. This algorithm also only works on square matrices with n being a power of 2. This algorithm is based on the observation that, for A X B = C. We can calculate C11, C12, C21 and C22 as,

@@ -2104,23 +2106,27 @@ Using the master's theorem
- -
-

7.2. Sorting algorithms

-
-
-

7.2.1. In place vs out place sorting algorithm

-
+
+

7. Lecture 7

+
+
+
+

7.1. Sorting algorithms

+
+
+
+

7.1.1. In place vs out place sorting algorithm

+

If the space complexity of a sorting algorithm is \(\theta (1)\), then the algorithm is called in place sorting, else the algorithm is called out place sorting.

-
-

7.2.2. Bubble sort

-
+
+

7.1.2. Bubble sort

+

Simplest sorting algorithm, easy to implement so it is useful when number of elements to sort is small. It is an in place sorting algorithm. We will compare pairs of elements from array and swap them to be in correct order. Suppose input has n elements.

@@ -2201,12 +2207,12 @@ Recursive time complexity : \(T(n) = T(n-1) + n - 1\)
-
-

8. Lecture 8

+
+

8. Lecture 8

-
-

8.1. Selection sort

+
+

8.1. Selection sort

It is an inplace sorting technique. In this algorithm, we will get the minimum element from the array, then we swap it to the first position. Now we will get the minimum from array[1:] and place it in index 1. Similarly, we get minimum from array[2:] and then place it on index 2. We do till we get minimum from array[len(array) - 2:] and place minimum on index [len(array) - 2]. @@ -2228,8 +2234,8 @@ It is an inplace sorting technique. In this algorithm, we will get the minimum e

-
-

8.1.1. Time complexity

+
+

8.1.1. Time complexity

The total number of comparisions is, @@ -2246,8 +2252,8 @@ Therefore the time complexity in all cases is, \[ \text{Time complexity} = \thet

-
-

8.2. Insertion sort

+
+

8.2. Insertion sort

It is an inplace sorting algorithm. @@ -2279,8 +2285,8 @@ It is an inplace sorting algorithm.

-
-

8.2.1. Time complexity

+
+

8.2.1. Time complexity

Best Case : The best case is when input array is already sorted. In this case, we do (n-1) comparisions and no swaps. The time complexity will be \(\theta (n)\) @@ -2303,8 +2309,8 @@ Total time complexity becomes \(\theta \left( 2 \frac{n(n-1)}{2} \right)\), whic

-
-

8.3. Inversion in array

+
+

8.3. Inversion in array

The inversion of array is the measure of how close array is from being sorted. @@ -2418,8 +2424,8 @@ Total number of inversions = 1 + 2 = 3

-
-

8.3.1. Relation between time complexity of insertion sort and inversion

+
+

8.3.1. Relation between time complexity of insertion sort and inversion

If the inversion of an array is f(n), then the time complexity of the insertion sort will be \(\theta (n + f(n))\). @@ -2428,8 +2434,8 @@ If the inversion of an array is f(n), then the time complexity of the insertion

-
-

8.4. Quick sort

+
+

8.4. Quick sort

It is a divide and conquer technique. It uses a partition algorithm which will choose an element from array, then place all smaller elements to it's left and larger to it's right. Then we can take these two parts of the array and recursively place all elements in correct position. For ease, the element chosen by the partition algorithm is either leftmost or rightmost element. @@ -2451,8 +2457,8 @@ As we can see, the main component of this algorithm is the partition algorithm.

-
-

8.4.1. Lomuto partition

+
+

8.4.1. Lomuto partition

The partition algorithm will work as follows: @@ -2488,9 +2494,12 @@ For an array of size n, the number ofcomparisions done by this algorithm

-
-

8.4.2. Time complexity of quicksort

+
+

8.4.2. Time complexity of quicksort

+

+In quick sort, we don't have a fixed recursive relation. The recursive relations differ for different cases. +

  • Best Case : The partition algorithm always divides the array to two equal parts. In this case, the recursive relation becomes \[ T(n) = 2T(n/2) + \theta (n) \] @@ -2505,8 +2514,356 @@ Again, \(\theta (n)\) is the time complexity for creating partition.
    Using master's theorem \[ T(n) = \theta (n^2) \]
    • + +
  • Average Case : The average case is closer to the best case in quick sort rather than to the worst case.
    • +
+

+
+To get the average case, we will consider a recursive function for number of comparisions \(C(n)\). +
+For the function \(C(n)\), there are \(n-1\) comparisions for the partition algorithm. +
+Now, suppose that the index of partition is i. +
+This will create two recursive comparisions \(C(i)\) and \(C(n-i-1)\). +
+i can be any number between 0 and n-1, with each case being equally probable. So the average number of comparisions for \(C(n)\) will be +\[ \frac{1}{n} \sum_{i=0}^{n-1} \left( C(i) + C(n-i-1) \right) \] +Therefore, total number of comparisions for input size n will be, +\[ C(n) = \left( n-1 \right) + \frac{1}{n} \sum_{i=0}^{n-1} \left( C(i) + C(n-i-1) \right) \] +Solving the above recurrance relation will give us, +\[ C(n) \approx 2\ n\ ln(n) \] +\[ C(n) \approx 1.39\ n\ log_2(n) \] +Therefore, the time complexity in average case becomes, +\[ T(n) = \theta (n\ log_2(n)) \] +

+
+
+ +
+

8.4.3. Number of comparisions

+
+

+The number of comparisions in quick sort for, +

+
    +
  • Worst Case : \[ \text{Number of comparisions} = \frac{n(n-1)}{2} \]
    • +
+
+
+
+ +
+

8.5. Merging two sorted arrays (2-Way Merge)

+
+

+Suppose we have two arrays that are already sorted. The first array has n elements and the second array has m elements. +
+The way to merge them is to compare the elements in a sequence between the two arrays. We first add a pointer to start of both arrays. The element pointed by the pointers are compared and the smaller one is added to our new array. Then we move pointer on that array forward. These comparisions are repeated until we reach the end of one of the array. At this point, we can simply append all the elements of the remaining array. +

+ +
+
int *merge(int a[], int n, int b[], int m){
+  int *c = malloc((m+n) * sizeof(int));
+  int i = 0; int j = 0;
+  int k = 0;
+
+  while (i !=  n && j != m) {
+    if ( a[i] > b[j] ) { c[k++] = b[j++]; } else { c[k++] = a[i++]; };
+  }
+
+  while (i != n) {
+    c[k++] = a[i++];
+  }
+
+  while (j != m) {
+    c[k++] = b[j++];
+  }
+
+  return c;
+}
+
+
+ +
    +
  • The maximum number of comparisions to merge the arrays is (m + n - 1).
    • +
  • The minimum number of comparisions to merge the arrays is either m or n. Depending of which one is smaller.
    • +
+
+
+ +
+

8.6. Merging k sorted arrays (k-way merge)

+
+

+k-way merge algorithms take k different sorted arrays and merge them into a single single array. The algorithm is same as that in two way merge except we need to get the smallest element from the pointer on k array's and then move it's corresponding pointer. +

+
+
+ +
+

8.7. Merge sort

+
+

+Merge sort is a pure divide and conquer algorithm. In this sorting algorithm, we merge the sorted sub-arrays till we get a final sorted array.
+The algorithm will work as follows : +

+
    +
  1. Divide the array of n elements into n subarrays, each having one element.
    2. +
  2. Repeatdly merge the subarrays to form merged subarrays of larger sizes until there is one list remaining.
    3. +
+ +

+For divide and conquer steps: +

+
    +
  • Divide : Divide the array from the middle into two equal sizes.
    • +
  • Conquer : Call merge sort recursively on the two subarrays
    • +
  • Combine : Merge the sorted array
    • +
+ +

+The algorithm works as follows (this isn't real c code) +

+
+
// A function that will merge two sorted arrays
+int[] merge(int first[], int second[]);
+
+int[] merge_sort(int array[], int left, int right){
+  if(left < right){
+    int mid = (left + right) / 2;
+    int sorted_first[] = merge_sort(array[], left, mid);
+    int sorted_second[] = merge_sort(array[], mid + 1, right);
+
+    return merge(sorted_first, sorted_second);
+  }
+}
+
+
+ +

+This algorithm is often used in languages which have great support for linked lists, for example lisp and haskell. For more traditional c-like languages, often quicksort is easier to implement. +
+An implementation in C language is as follows. +

+ +
+
// buffer is memory of size equal to or bigger than size of array
+// buffer is used when merging the arrays
+void merge_sort(int array[], int left, int right, int buffer[]){
+  if(left < right){
+    // Divide part
+    int mid = ( left + right ) / 2;
+
+    // Conquer part
+    merge_sort(array,left, mid, buffer);
+    merge_sort(array, mid + 1, right, buffer);
+
+    // Combine part : Merges the two sorted parts
+    int i = left; int j = mid + 1; int k = 0;
+    while( i != (mid+1) && j != (right+1) ){
+      if(array[i] < array[j]) { buffer[k++] = array[i++];  } else { buffer[k++] = array[j++]; }
+    }
+
+    while(i != (mid+1))
+      buffer[k++] = array[i++];
+
+    while(j != (right+1))
+      buffer[k++] = array[j++];
+
+    for(int x = left; x <= right; x++)
+      array[x] = buffer[x - left];
+  }
+}
+
+
+
+ +
+

8.7.1. Time complexity

+
+

+Unlike quick sort, the recurrance relation is same for merge sort in all cases. +
+Since divide part divides array into two equal sizes, the input size is halfed (i.e, T(n/2) ). +
+In conquer part, there are two calls so 2.T(n/2) is added to time complexity. +
+The cost for merging two arrays of size n/2 each is either n-1 of n/2. That is to say that time complexity to merge two arrays of size n/2 each is always \(\theta (n)\). Thus, the final recurrance relation is +\[ T(n) = 2.T(n/2) + \theta (n) \] +Using the master's theorem. +\[ T(n) = \theta (n.log_2n) \] +

+
+
+ +
+

8.7.2. Space complexity

+
+

+As we can see in the C code, the space complexity is \(\theta (n)\) +

+
+
+
+ +
+

8.8. Stable and unstable sorting algorithms

+
+

+We call sorting algorithms unstable or stable on the basis of whether they change order of equal values. +

+
    +
  • Stable sorting algorithm : a sorting algorithm that preserves the order of the elements with equal values.
    • +
  • Unstable sorting algorithm : a sorting algorithm that does not preserve the order of the elements with equal values. +
    • +
+

+This is of importance when we store data in pairs of keys and values and then sort data using the keys. So we may want to preserve the order in which the entries where added. +
+Example, suppose we add (key, value) pairs as: +

+
+(2, v1), (1, v2), (3, v3), (1, v1), (2, v4), (3, v2)
+
+ +

+Now, if we sort using the keys a sorting algorithm which is stabe will preserve the order of elements with equal keys. So output is always +

+
+(1, v2), (1, v1), (2,v1), (2, v4), (3, v3), (3, v2)
+
+

+i.e, the order of keys with same values is preserved. +
+Whereas an unstable sorting algorithm will sort without preserving the order of key values. +

+
+
+ +
+

8.9. Non-comparitive sorting algorithms

+
+

+Sorting algorithms which do not use comparisions to sort elements are called non-comparitive sorting algorithms. These tend to be faster than comparitive sorting algorithms. +

+
+ +
+

8.9.1. Counting sort

+
+
    +
  • Counting sort only works on integer arrays
    • +
  • Couting sort only works if all elements of array are non-negative, i.e, elements are only allowed to be in range [0,k] .
+ +
+
//* The input array is sorted and result is stored in output array *//
+//* max is the largest element of the array *//
+void counting_sort(int input[], int max ,int output[]){
+  // count array should have a size greater than or equal to (max + 1)
+  int count[max + 1];
+  // initialize count array to zero, can also use memset
+  for(int i = 0; i < max+1; i++) count[i] = 0;
+
+  // i from 0 to len(array) - 1
+  // this loop stores number of elements equal to i in count array
+  for(int i = 0; i < len(array); i++)
+    count[input[i]] = count[input[i]] + 1;
+
+  // i from 1 to max
+  // this loop stores number of elements less that or equal to i in count array
+  for(int i = 1; i <= max; i++)
+    count[i] = count[i] + count[i - 1];
+
+  // i from len(array) - 1 to 0
+  for(int i = len(array) - 1; i >= 0; i--){
+    count[input[i]] = count[input[i]] - 1;
+    output[count[input[i]]] = input[i];
+  }
+}
+
+
+ +
    +

  • +Time complexity : Since there are only simple loops and arithmetic operations, we can get time complexity by considering the number of times loops are executed. +

    + +

    +\[ \text{Number of times loops are executed} = n + (max - 1) + n \] +\[ \text{Where, } n = len(array) \text{ i.e, the input size} \] +

    + +

    +Therefore, +\[ \text{Number of times loops are executed} = 2n + max - 1 \] +\[ \text{Time complexity} = \theta (n + max) \] +

    • +
+
+ +
+

8.9.2. Radix sort

+
+

+In radix sort, we sort using the digits, from least significant digit (lsd) to most significant digit (msd). In other words, we sort digits from right to left. The algorithm used to sort digits should be a stable sorting algorithm. +

+ + +
+

radix-sort.png +

+
+ +

+For the following example, we will use the bubble sort since it is the easiest to implement. But, for best performance, radix sort is paired with counting sort. +

+ +
+
// d = 0, will return digit at unit's place
+// d = 1, will return digit at ten's place
+// and so on.
+int get_digit(int n, int d){
+  assert(d >= 0);
+  int place = (int) pow(10, d);
+  int digit = (n / place) % 10;
+  return digit;
+}
+
+// bubble sort the array for only digits of the given place
+// d = 0, unit's place
+// d = 1, ten's place
+// and so on.
+void bubble_sort_digit(int array[], int d){
+  for(int i = len(array); i >= 1; i--){
+    for(int j = 0; j < i; j++){
+      if(get_digit(array[j], d) > get_digit(array[j + 1], d))
+        array[j], array[j + 1] = array[j + 1], array[j];
+    }
+  }
+}
+
+void radix_sort(int array[], int no_of_digits){
+  for(int i = 0; i < no_of_digits ; i++){
+    bubble_sort_digit(array, i );
+  }
+}
+
+
+ +
    +
  • Time complexity : \[ \text{Time Complexity} = \theta (d.(n + max)) \] +Where, d = number of digits in max elemet, and +
    +radix sort is paired with counting sort.
    • +
+
+
+ +
+

8.9.3. Bucket sort

diff --git a/main.tsk b/main.tsk index e7b4814..e1d6aa4 100644 --- a/main.tsk +++ b/main.tsk @@ -2,6 +2,6 @@ #do emacs --script export.el -*Remove Intermediate +*Remove intermediate #do -rm main.html~ \ No newline at end of file +rm main.html~