@ -39,3 +39,28 @@ Since algorithms are finite, they have *bounded time* taken and *bounded space*
+ The Big Oh notation is used to define the upper bound of an algorithm.
+ Given a non negative funtion f(n) and other non negative funtion g(n), we say that $f(n) = O(g(n)$ if there exists a positive number $n_0$ and a positive constant $c$, such that \[ f(n) \le c.g(n) \ \ \forall n \ge n_0 \]
+ So if growth rate of g(n) is greater than or equal to growth rate of f(n), then $f(n) = O(g(n))$.
** Omega Notation [ $\Omega$ ]
+ It is used to shown the lower bound of the algorithm.
+ For any positive integer $n_0$ and a positive constant $c$, we say that, $f(n) = \Omega (g(n))$ if \[ f(n) \ge c.g(n) \ \ \forall n \ge n_0 \]
+ So growth rate of $g(n)$ should be less than or equal to growth rate of $f(n)$
*Note* : If $f(n) = O(g(n))$ then $g(n) = \Omega (f(n))$
** Theta Notation [ $\theta$ ]
+ If is used to provide the asymptotic *equal bound*.
+ $f(n) = \theta (g(n))$ if there exists a positive integer $n_0$ and a positive constants $c_1$ and $c_2$ such that \[ c_1 . g(n) \le f(n) \le c_2 . g(n) \ \ \forall n \ge n_0 \]
+ So the growth rate of $f(n)$ and $g(n)$ should be equal.
*Note* : So if $f(n) = O(g(n))$ and $f(n) = \Omega (g(n))$, then $f(n) = \theta (g(n))$
** Little-Oh Notation [o]
+ The little o notation defines the strict upper bound of an algorithm.
+ We say that $f(n) = o(g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) < c.g(n) \ \ \forall n \ge n_0 \]
+ Notice how condition is <, rather than $\le$ which is used in Big-Oh. So growth rate of $g(n)$ is strictly greater than that of $f(n)$.
** Little-Omega Notation [ $\omega$ ]
+ The little omega notation defines the strict lower bound of an algorithm.
+ We say that $f(n) = \omega (g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) > c.g(n) \ \ \forall n \ge n_0 \]
+ Notice how condition is >, rather than $\ge$ which is used in Big-Omega. So growth rate of $g(n)$ is strictly less than that of $f(n)$.
+ It is used to shown the lower bound of the algorithm.
+ For any positive integer $n_0$ and a positive constant $c$, we say that, $f(n) = \Omega (g(n))$ if \[ f(n) \ge c.g(n) \ \ \forall n \ge n_0 \]
+ So growth rate of $g(n)$ should be less than or equal to growth rate of $f(n)$
*Note* : If $f(n) = O(g(n))$ then $g(n) = \Omega (f(n))$
** Theta Notation [ $\theta$ ]
+ If is used to provide the asymptotic *equal bound*.
+ $f(n) = \theta (g(n))$ if there exists a positive integer $n_0$ and a positive constants $c_1$ and $c_2$ such that \[ c_1 . g(n) \le f(n) \le c_2 . g(n) \ \ \forall n \ge n_0 \]
+ So the growth rate of $f(n)$ and $g(n)$ should be equal.
*Note* : So if $f(n) = O(g(n))$ and $f(n) = \Omega (g(n))$, then $f(n) = \theta (g(n))$
** Little-Oh Notation [o]
+ The little o notation defines the strict upper bound of an algorithm.
+ We say that $f(n) = o(g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) < c.g(n) \ \ \forall n \ge n_0 \]
+ Notice how condition is <, rather than $\le$ which is used in Big-Oh. So growth rate of $g(n)$ is strictly greater than that of $f(n)$.
** Little-Omega Notation [ $\omega$ ]
+ The little omega notation defines the strict lower bound of an algorithm.
+ We say that $f(n) = \omega (g(n))$ if there exists positive integer $n_0$ and positive constant $c$ such that, \[ f(n) > c.g(n) \ \ \forall n \ge n_0 \]
+ Notice how condition is >, rather than $\ge$ which is used in Big-Omega. So growth rate of $g(n)$ is strictly less than that of $f(n)$.
* Comparing Growth rate of funtions
** Applying limit
To compare two funtions $f(n)$ and $g(n)$. We can use limit
The addition of matricies of size (n X n) takes time $\theta (n^2)$, therefore, for computation of C_11 will take time of $\theta \left( \left( \frac{n}{2} \right)^2 \right)$, which is equals to $\theta \left( \frac{n^2}{4} \right)$. Therefore, computation time of C_11, C_12, C_21 and C_22 combined will be $\theta \left( 4 \frac{n^2}{4} \right)$, which is equals to $\theta (n^2)$.
\\
There are 8 recursive calls in this function with MatrixMul(n/2), therefore, time complexity will be
\[ T(n) = 8T(n/2) + \theta (n^2) \]
Using the *master's theorem*
\[ T(n) = \theta (n^{log_28}) \]
\[ T(n) = \theta (n^3) \]
** Strassen's algorithm
Another, more efficient divide and conquer algorithm for matrix multiplication. This algorithm also only works on square matrices with n being a power of 2. This algorithm is based on the observation that, for A X B = C. We can calculate C_11, C_12, C_21 and C_22 as,
\[ \text{C_11 = P_5 + P_4 - P_2 + P_6} \]
\[ \text{C_12 = P_1 + P_2} \]
\[ \text{C_21 = P_3 + P_4} \]
\[ \text{C_22 = P_1 + P _5 - P_3 - P_7} \]
Where,
\[ \text{P_1 = A_11 X (B_12 - B_22)} \]
\[ \text{P_2 = (A_11 + A_12) X B_22} \]
\[ \text{P_3 = (A_21 + A_22) X B_11} \]
\[ \text{P_4 = A_22 X (B_21 - B_11)} \]
\[ \text{P_5 = (A_11 + A_22) X (B_11 + B_22)} \]
\[ \text{P_6 = (A_12 - A_22) X (B_21 + B_22)} \]
\[ \text{P_7 = (A_11 - A_21) X (B_11 + B_12)} \]
This reduces number of recursion calls from 8 to 7.
#+BEGIN_SRC
Strassen(A, B, n):
If n == 2 {
return A X B
}
Else{
Break A into four parts A_11, A_12, A_21, A_22, where A = [[ A_11, A_12],
[ A_21, A_22]]
Break B into four parts B_11, B_12, B_21, B_22, where B = [[ B_11, B_12],
[ B_21, B_22]]
P_1 = Strassen(A_11, B_12 - B_22, n/2)
P_2 = Strassen(A_11 + A_12, B_22, n/2)
P_3 = Strassen(A_21 + A_22, B_11, n/2)
P_4 = Strassen(A_22, B_21 - B_11, n/2)
P_5 = Strassen(A_11 + A_22, B_11 + B_22, n/2)
P_6 = Strassen(A_12 - A_22, B_21 + B_22, n/2)
P_7 = Strassen(A_11 - A_21, B_11 + B_12, n/2)
C_11 = P_5 + P_4 - P_2 + P_6
C_12 = P_1 + P_2
C_21 = P_3 + P_4
C_22 = P_1 + P_5 - P_3 - P_7
C = [[ C_11, C_12],
[ C_21, C_22]]
return C
}
#+END_SRC
This algorithm uses 18 matrix addition operations. So our computation time for that is $\theta \left(18\left( \frac{n}{2} \right)^2 \right)$ which is equal to $\theta (4.5 n^2)$ which is equal to $\theta (n^2)$.
\\
There are 7 recursive calls in this function which are Strassen(n/2), therefore, time complexity is
\[ T(n) = 7T(n/2) + \theta (n^2) \]
Using the master's theorem
\[ T(n) = \theta (n^{log_27}) \]
\[ T(n) = \theta (n^{2.807}) \]
+ /*NOTE* : The divide and conquer approach and strassen's algorithm typically use n == 1 as their terminating condition since for multipliying 1 X 1 matrices, we only need to calculate product of the single element they contain, that product is thus the single element of our resultant 1 X 1 matrix./
The addition of matricies of size (n X n) takes time $\theta (n^2)$, therefore, for computation of C_11 will take time of $\theta \left( \left( \frac{n}{2} \right)^2 \right)$, which is equals to $\theta \left( \frac{n^2}{4} \right)$. Therefore, computation time of C_11, C_12, C_21 and C_22 combined will be $\theta \left( 4 \frac{n^2}{4} \right)$, which is equals to $\theta (n^2)$.
\\
There are 8 recursive calls in this function with MatrixMul(n/2), therefore, time complexity will be
\[ T(n) = 8T(n/2) + \theta (n^2) \]
Using the *master's theorem*
\[ T(n) = \theta (n^{log_28}) \]
\[ T(n) = \theta (n^3) \]
** Strassen's algorithm
Another, more efficient divide and conquer algorithm for matrix multiplication. This algorithm also only works on square matrices with n being a power of 2. This algorithm is based on the observation that, for A X B = C. We can calculate C_11, C_12, C_21 and C_22 as,
\[ \text{C_11 = P_5 + P_4 - P_2 + P_6} \]
\[ \text{C_12 = P_1 + P_2} \]
\[ \text{C_21 = P_3 + P_4} \]
\[ \text{C_22 = P_1 + P _5 - P_3 - P_7} \]
Where,
\[ \text{P_1 = A_11 X (B_12 - B_22)} \]
\[ \text{P_2 = (A_11 + A_12) X B_22} \]
\[ \text{P_3 = (A_21 + A_22) X B_11} \]
\[ \text{P_4 = A_22 X (B_21 - B_11)} \]
\[ \text{P_5 = (A_11 + A_22) X (B_11 + B_22)} \]
\[ \text{P_6 = (A_12 - A_22) X (B_21 + B_22)} \]
\[ \text{P_7 = (A_11 - A_21) X (B_11 + B_12)} \]
This reduces number of recursion calls from 8 to 7.
#+BEGIN_SRC
Strassen(A, B, n):
If n == 2 {
return A X B
}
Else{
Break A into four parts A_11, A_12, A_21, A_22, where A = [[ A_11, A_12],
[ A_21, A_22]]
Break B into four parts B_11, B_12, B_21, B_22, where B = [[ B_11, B_12],
[ B_21, B_22]]
P_1 = Strassen(A_11, B_12 - B_22, n/2)
P_2 = Strassen(A_11 + A_12, B_22, n/2)
P_3 = Strassen(A_21 + A_22, B_11, n/2)
P_4 = Strassen(A_22, B_21 - B_11, n/2)
P_5 = Strassen(A_11 + A_22, B_11 + B_22, n/2)
P_6 = Strassen(A_12 - A_22, B_21 + B_22, n/2)
P_7 = Strassen(A_11 - A_21, B_11 + B_12, n/2)
C_11 = P_5 + P_4 - P_2 + P_6
C_12 = P_1 + P_2
C_21 = P_3 + P_4
C_22 = P_1 + P_5 - P_3 - P_7
C = [[ C_11, C_12],
[ C_21, C_22]]
return C
}
#+END_SRC
This algorithm uses 18 matrix addition operations. So our computation time for that is $\theta \left(18\left( \frac{n}{2} \right)^2 \right)$ which is equal to $\theta (4.5 n^2)$ which is equal to $\theta (n^2)$.
\\
There are 7 recursive calls in this function which are Strassen(n/2), therefore, time complexity is
\[ T(n) = 7T(n/2) + \theta (n^2) \]
Using the master's theorem
\[ T(n) = \theta (n^{log_27}) \]
\[ T(n) = \theta (n^{2.807}) \]
+ /*NOTE* : The divide and conquer approach and strassen's algorithm typically use n == 1 as their terminating condition since for multipliying 1 X 1 matrices, we only need to calculate product of the single element they contain, that product is thus the single element of our resultant 1 X 1 matrix./
Solving the above recurrance relation will give us,
\[ C(n) \approx 2\ n\ ln(n) \]
\[ C(n) \approx 1.39\ n\ log_2(n) \]
Therefore, the time complexity in average case becomes,
\[ T(n) = \theta (n\ log_2(n)) \]
** Number of comparisions
The number of comparisions in quick sort for,
+ Worst Case : \[ \text{Number of comparisions} = \frac{n(n-1)}{2} \]
* Merging two sorted arrays (2-Way Merge)
Suppose we have two arrays that are already sorted. The first array has *n* elements and the second array has *m* elements.
\\
The way to merge them is to compare the elements in a sequence between the two arrays. We first add a pointer to start of both arrays. The element pointed by the pointers are compared and the smaller one is added to our new array. Then we move pointer on that array forward. These comparisions are repeated until we reach the end of one of the array. At this point, we can simply append all the elements of the remaining array.
+ The maximum number of comparisions to merge the arrays is (m + n - 1).
+ The minimum number of comparisions to merge the arrays is either *m* or *n*. Depending of which one is smaller.
* Merging k sorted arrays (k-way merge)
k-way merge algorithms take k different sorted arrays and merge them into a single single array. The algorithm is same as that in two way merge except we need to get the smallest element from the pointer on k array's and then move it's corresponding pointer.
* Merge sort
Merge sort is a pure divide and conquer algorithm. In this sorting algorithm, we merge the sorted sub-arrays till we get a final sorted array.\\
The algorithm will work as follows :
1. Divide the array of n elements into *n* subarrays, each having one element.
2. Repeatdly merge the subarrays to form merged subarrays of larger sizes until there is one list remaining.
For divide and conquer steps:
+ *Divide* : Divide the array from the middle into two equal sizes.
+ *Conquer* : Call merge sort recursively on the two subarrays
+ *Combine* : Merge the sorted array
The algorithm works as follows (this isn't real c code)
#+BEGIN_SRC C
// A function that will merge two sorted arrays
int[] merge(int first[], int second[]);
int[] merge_sort(int array[], int left, int right){
if(left < right){
int mid = (left + right) / 2;
int sorted_first[] = merge_sort(array[], left, mid);
int sorted_second[] = merge_sort(array[], mid + 1, right);
return merge(sorted_first, sorted_second);
}
}
#+END_SRC
This algorithm is often used in languages which have great support for linked lists, for example lisp and haskell. For more traditional c-like languages, often quicksort is easier to implement.
\\
An implementation in C language is as follows.
#+BEGIN_SRC C
// buffer is memory of size equal to or bigger than size of array
// buffer is used when merging the arrays
void merge_sort(int array[], int left, int right, int buffer[]){
Unlike quick sort, *the recurrance relation is same for merge sort in all cases.*
\\
Since divide part divides array into two equal sizes, the input size is halfed (i.e, *T(n/2)* ).
\\
In conquer part, there are two calls so *2.T(n/2)* is added to time complexity.
\\
The cost for merging two arrays of size n/2 each is either *n-1* of *n/2*. That is to say that time complexity to merge two arrays of size n/2 each is always $\theta (n)$. Thus, the final recurrance relation is
\[ T(n) = 2.T(n/2) + \theta (n) \]
Using the master's theorem.
\[ T(n) = \theta (n.log_2n) \]
** Space complexity
As we can see in the C code, the space complexity is $\theta (n)$
* Stable and unstable sorting algorithms
We call sorting algorithms unstable or stable on the basis of whether they change order of equal values.
+ *Stable sorting algorithm* : a sorting algorithm that preserves the order of the elements with equal values.
+ *Unstable sorting algorithm* : a sorting algorithm that does not preserve the order of the elements with equal values.
\\
This is of importance when we store data in pairs of keys and values and then sort data using the keys. So we may want to preserve the order in which the entries where added.
i.e, the *order of keys with same values is preserved*.
\\
Whereas an unstable sorting algorithm will sort without preserving the order of key values.
* Non-comparitive sorting algorithms
Sorting algorithms which do not use comparisions to sort elements are called non-comparitive sorting algorithms. These tend to be faster than comparitive sorting algorithms.
** Counting sort
+ Counting sort *only works on integer arrays*
+ Couting sort only works if *all elements of array are non-negative*, i.e, elements are only allowed to be in range [0,k] .
#+BEGIN_SRC c
//* The input array is sorted and result is stored in output array *//
//* max is the largest element of the array *//
void counting_sort(int input[], int max ,int output[]){
// count array should have a size greater than or equal to (max + 1)
int count[max + 1];
// initialize count array to zero, can also use memset
for(int i = 0; i < max+1; i++) count[i] = 0;
// i from 0 to len(array) - 1
// this loop stores number of elements equal to i in count array
for(int i = 0; i < len(array); i++)
count[input[i]] = count[input[i]] + 1;
// i from 1 to max
// this loop stores number of elements less that or equal to i in count array
for(int i = 1; i <= max; i++)
count[i] = count[i] + count[i - 1];
// i from len(array) - 1 to 0
for(int i = len(array) - 1; i >= 0; i--){
count[input[i]] = count[input[i]] - 1;
output[count[input[i]]] = input[i];
}
}
#+END_SRC
+ *Time complexity* : Since there are only simple loops and arithmetic operations, we can get time complexity by considering the number of times loops are executed.
\[ \text{Number of times loops are executed} = n + (max - 1) + n \]
\[ \text{Where, } n = len(array) \text{ i.e, the input size} \]
Therefore,
\[ \text{Number of times loops are executed} = 2n + max - 1 \]
\[ \text{Time complexity} = \theta (n + max) \]
** Radix sort
In radix sort, we sort using the digits, from least significant digit (lsd) to most significant digit (msd). In other words, we sort digits from right to left. The algorithm used to sort digits *should be a stable sorting algorithm*.
[[./imgs/radix-sort.png]]
For the following example, we will use the bubble sort since it is the easiest to implement. But, for best performance, *radix sort is paired with counting sort*.
#+BEGIN_SRC c
// d = 0, will return digit at unit's place
// d = 1, will return digit at ten's place
// and so on.
int get_digit(int n, int d){
assert(d >= 0);
int place = (int) pow(10, d);
int digit = (n / place) % 10;
return digit;
}
// bubble sort the array for only digits of the given place
