Lecture 1: Algorithms
Knuth’s characterization:
-​ Input specified → What do we work on?
-​ Output specified → What do we deliver?
-​ Definiteness → What do we do at each step? You know what to do at each step.
-​ Effectiveness → When the steps are properly applied, the result should be what is
expected.
-​ Finiteness → the algorithms finishes
Input is specified, it takes 2 integers. The
output is also specified, 1 integer. It is all
python so it is well defined. It is in a
programming language so it knows what
to do at each step. There is an end
condition.
Algorithm nesting: calling a function inside a function.
Data structure is thet way to organise data:
-​ Data is stored in memery as numbers
-​ Data is stored as objects of a programming language
-​ Data is manipulated through functions or object methods.
-​ Data is organized by objects relations
Data structures are represented by class, and operation by methods of the class.
Complexity tries to answer how many times more does the algorithm take, when the data
size increases by X amount.
The complexity of SlowSort is O(N**2).
For complexity, only the largest “determinant matters”. For example: O(N**2 + 6N + 19)
converts to O(N**2) when N goes to infinity.
Most sorting algorithms have a complexity of O(N * log(N)).
The same applies to memory.
Lecture 2: Sequences
Grading algorithm:
Take the top of the todo list, grade it and
put it in graded. Continue until todo is
empty.
What is the data structure? This is a stack.
The last paper put in the pile, is the first to
be removed.
Sequence: an enumerated object where
repetition is allowed and order matters.
Collection: one sequence contains many objects.
Enumerated: each object has a positive integer index.
Repetitions: one object can appear many times.
Order: there is a first, a second, etc.
List:
Build, get its length, get an object at index, change object at index, insert object at index,
search index.
Inserting at the start is the worst-case scenario for a Python list.
Deleting at the start is the worst-case scenario for a Python list.
Deque:​
Double ended queue.
You have to choose the data structure by how you are going to use it.
Stack: LIFO
-​ Pop to remove at the start
-​ Push to insert at the end
Queue: FIFO
-​ Pop to remove at the start
-​ Append to insert at the end
Lecture 3: Algorithms on Sequences
Guess & Check
Given a problem, and a potential solution. Check whether the actual solution is in the list of
potential solutions. Returns the solution if found, otherwise it returns nothing.
Guess & Check v2
A polynom is a continuous function, if no solution is found, it still gives some information
regarding the solution.
Complexity: on average the number will be in the middle so;½ N which converts to O(N).
Bisection:
You want to find something that is a sorted list. Because they are sorted in ascending order,
you can zoom in very fast. It works usign pivot points.
Complexity. N = number of guesses, m = degree of polynomial.
O(m * log(N))
Approximation: find the zeros of a complication
function.
We can approximate where the zeros are.
F(x) must be continuous, and the derivative must
also be continuous.
Idea: changing the search space at every step,
the algorithm starts again with a new list of
guesses, active intervention on the search space.
There is also a stack “hidden” in the algorithm.
Take away:​
Problem can be reduced to:
-​ Guess & Check: try all the possibilities
-​ Bisection: smae, but reduces the number of possibilities quickly.
-​ Approximation: uses Bisection, searches in-between the possibilities.
Lecture 4: Algorithms on Sequences: Sorting
In a sorted list, searching takes a lot less time.
Given a sequence of comparable objects, after sorting it is ordered so Si+1 > Si. Also called
total order.
Numbesr and strings are naturally comparable, but not everything can be ordered.
Why sorting?
-​ Process by priority, by size, by alphabetical order…..
-​ Search
-​ Vanilla search is O(N)
-​ But when it is sorted O(log(N))
-​ Spend some time ONCE to sort
-​ Spen little times MANY TIMES to search
Selection sort:
Creating a copy C of L → O(len(L))
Find the minimum value in C → O(len(C))
Append it to S → O(1)
Remove it from C → O(len(C))
The list C changes every time, it keeps reducing. However,
in the end it still grows as N**2.
Pigeonhole sort:
Only works on integers. You estimate
that all numbers are between 0 - 10,
and count how many times you see
each number, and create a sorted list
afterwards.
It works because there is a finite
amount of real numbers.
Using information about the data:
-​ Pigeonhole is much more
efficient for lists of integers.
Quicksort:
Works by selecting a pivot point.
C(N) = O (N * log(N))
What did we learn?
-​ Quicksort is much faster than selection sort
Lecture 5: Sets
Set: collection of unique elements without
order.
Dictionary: set of keys, each on eassociated
to a value.
Key point: how can we find if a value is unique
in the fastest way possible?
Hash Tables:
2 objects that are equal have the same hash.
2 objects that are not equal are very unlikely
to have the same hash.
2 objects with a small difference are very likely to have very different hashes.
Most hash-funciton are a one-way function. It is easy to compute hash(x) given x, but very
hard to retrieve x given hash(x).
Collisions: when two objects that are not equal, but have the same hash value.
In reality there is not enough memory.
If an object is hashable, this means it is immutable, and the object can’t be modified. You
want the hash to be constant throughout the life of an object.
Dictionaries:
Only hashable objects can be used as keys.
No constraint on values.
Most important point: you have a set of keys, and they are unique. You achieve this by using
a set, and using a hash table.
Caching: spend a little bit of memory to save (a lot of) time.
Lecture 6: Matrices
Matrices and AI.
Why do we focus so much on matrixes? It is the key to everything that we do with AI.
Each output layer has as many parameters as the inputs, bcause first it does a weighted
sum of the inputs. Going from one layer to another is a matrix multiplication:
Outputs = Weights x Inputs.
A logistic regression is a matrix multiplication.
Attention mechanism: getting an input and doing 2 matrix multiplication:
-​ Transfor the input and project them into another vector space
-​ After that you have an attention matrix, because of the training, the matrix is full of
zero’s.
Matrix in Python:
How can we get a matrix represented as an object → list of lists.
Numbers are all over the in memory, which makes it very inefficient to retrieve.
The usual representation is targeting to have a contiguous array of numbers: the matrix
numbers are stored in one long row, if needed everything is collected at once, which takes
less time.
Multidimensional arrays, Numpy has support for this, like a 4 dimension matrix.
Example: pixel of the screen, each pixel has 3 colours: red, green, and blue. So a gif made
of 3 images of 7x2 pixels with RGB colors. So in total the 4-D array is 3x7x2x3 = 126
elements.
!!! order matters !!!
Dense or Sparse?
-​ Few zeros = DENSE
-​ Mostly zeros = SPARSE
In real life, sparse matrixes are made of 99.5% of zeros. There is no clear threshold for
which dense and sparse matrixes are qualified, it is mostly the extremes.
Matrix Multiplication:
Dense matrix multiplication:
There are inbedded for-loops and they all have the same number of repetitions. This leads to
a complexity of O(N**3).
Why is the computer doing better than predicted? 8000 should be close to 3 minutes but it
was 5.2 seconds? If it is multiplied by 2, it takes 82 seconds, where it should take around 10
minutes. Numpy implements parallel computing, when computing the dot product of one row
and column, other cores are free to compute the dot products of different rows and columns.
This is the secret of “beating” the time complexity of N**3.
The time complexity has been beaten by an algorithm called Strassen which is O(N**2.81).
Last year it was done by an algorithm with complexity O(N**2.37), but it is unusable,
because constant and multiplication factors are hidden.
Sparse Matrix
Made of mostly zeros.
Example: user product review.
There are a certain number of users and products. 1 million products, 10 million users. How
many products do a user actually grade and review?
This is a matrix of 10**13 cells and 40TB in memory.
If you focus only on the non-zeros, it would be 200MB in memory, and 5x10**7 cells.
The idea is that you only store cells which contain non-zero data. We are going to store a list
of coordinates with corresponding values, if it is not in the list → it is a zero.
The linear search can be improved by coming from both sides which divides the time by 2. A
better way is using Pigeonhole because it is made of integers. Pigeonhole is used to
produce a sorted list.
List: sort it and search using Bisection.
Take away:
You have to find the right data structure and the right data preparation.
Lecture 7: Graphs
Graphs
A lot of problems are related to graphs. Sometimes it may not look like it, but most problems
can be translated into a graph.
Nodes: usually represented as a dot. Each node is unique. Nodes can also be called
vertices.
Edges: are the links between the nodes.
Undirected Graph: it has no direction, there are no arrows drawn on the edges. Edges can
be used both ways.
There is a source and a target.
If you are in node 3, it is impossible to go to node 1. Once you arrive in 2, you cannot leave
since there are no edges out of 2.
Any object can be a node as long as it is hashable.
Nodes can have attributes:
In an undirected graph, the degree of a node are the edges connected to that node. In a
directed graph, the in-degree is the number of edges arriving on a node, and the out-degree
is the number of edges leaving a node.
Walk: a route of nodes by following the edges.
Cycle: a walk that comes back to its starting point.
A eulerian pass is when each edge is visited once. This is only possible if each node has an
even degree.
Lecture 8: Graph traversal/ shortest path
Look at the graph from above and look where you can go when you start in 1.
Breadth-First Search:
Start from a node, visit the neigbors, once they are visited, you will visit their neighbors.
Keep track of which nodes are already visited. The notes are visited by layers.
Popping the first item of a queue with N nodes → O(1)
If w is not in the visited nodes → O(1)
​
Append W to Q → O(1)
V = number of nodes
E = number of edges
The total complexity is O(V) + O(sum(neighbors)) = O(V) + O(E) = O(V + E).
Shortest path:
A graph has some cost/ distance to each edge. A weighted walk is the sum of the values of
teh edges.
Dijksta applies to distance graphs, it has a weight attributed to edges which is strictly
positive, like distance and time, you do not travel by negative time or distance.
Put all the nodes into a priority queue, and get them out one by one and update their priority.
At every round we pick out the node with the lowest priority(distance to the starting node).
The main idea is that when a node is removed from the queue the shortest path has been
found. Little by little you establish the shortest path for each node,
Source = 1.
Queue = (1,0) (2, +inf) (3, +inf) ……
Pop node 1, it is the one with the lowest priority, there is no shorter path from 1 to 1 than 0.
Node 1 has two neighbors [2, 3].
W = 2, priority(1) = dist(1,2) = 20.1
20.1 is smaller than infinity, to update 2: 20.1
Queue = (2, 20.1) (3, +inf) (4, +inf)
W=3
5.1 is smaller than infinity so update 3: 5.1
Queue = (2, 20.1) (3, 5.1) (4, +inf)
Pop node 3, because it has the shortes distance from 1, since no path is shorter than 5.1
from node 1 to node 3.
Neighbors = [2, 4]
Queue = (2, 20.1) (4, +inf)
W=2
prioriry (3) + distance (3,2) = 8.3
Queue = (2, 8.3) (4, +inf)
W=4
5.1 + 4.6 is smaller than infinite
Queue = (2, 8.3) (4, 9.7)
Pop 2, it has no neighbors, so nothing is updated. Pop 4, it has 1 neighbor but it is not in the
queue and the queue is empty and you are done.
Lecture 9: Applications of Graph Coloring
Example: scheduling exam at the UVA.
There are different exams, any exam can take place in any room, but only one exam in a
room. Each has a start and end time. How many rooms do we need?
History: 12-14, Math: 11-13, French: 10-12, Python: 14-16, you have 3 rooms.
History and math overlap, French and math as well, so they can share the same room.
This is a graph coloring problem. Each node is assigned a color, but adjacent nodes need to
have different colors, so the goal is to minimize the amount of colors used.
Sudoku:
To color it, we are going to use a recursive coloring algorithm. There is a graph made of
nodes with numbers 0, ….. , node + 1.
If the node is already colored, you call recursively N+1.
if it is not colored, you have to identify which color is safe to use (which colors the neighbors
are not using).
Best case: the first color choses for each node is the solution. → O(V + E)
Worst case: evertime you check a safe color it is the last color and you only discover it after
calling the next and the first node that is wrong is the last node in the graph → O(phi**V).
Important point: if you don’t know that you will find the solution fast, be ready to expect a
long run time.
Greedy algorithm: Welsh-Powell
It is greedy because it allocates colors on the fly. This means you do not have control over
the number of colors, but it is very fast. Complexity O(V + E).
!! It may not look like it, but a lot of (business) problems are graphs !!
Lecture 10: Searching
Web Search:
When you look something up on Google, the first 3 results are mostly very useful.
Web pages contain hyperlinks to other pages. The web is just a very large network of
webpages. These links do not appear as random, someone has decided to put the link there.
Some nodes in the network have a high in-degree. If a page has a high in-degree, it
indicates that it is an import page. This also means that if a page has a high in-degree to
another page, it gives the other page some “credit”.
PageRank:
Drop a walker on a random node, and take note of
which nodes are visitsed. If you repeat this many
times, you will see that the importance of the nodes
starts to appear.
So a High PageRank = authoritative pages.
At first every node has the same amount of
importance. At each round, your importance is
distributed to your out-edges, and you receive some
share of the importance of those who have an edge towards you. After a while there is a
stable state which indicates your importance.
Each node starts with the same
importance. There is a transition
matrix, we read that node 1 gives ½
to node 2 and ½ to node 3.
PageRank with random surfer
Now not everything is being given away, but 90% is split between neighbors, and 10% is
split between non-neighbors.
When a score is created for a page, the idea is that the search engine creates a score for all
the pages, and it will return the list with the top priorities.
Link farming: creating fake webpages that reference the actual page, to push it up in the
rankings.
Pagerank also works for text summarization.
Lecture 11: Minimum Spanning Tree
We want to link offices all over the Netherlands. You want to build a network that connects
every office, but there are also costs. However, not every pair needs to be interconnected.
How can you pay the least amount of money but still have all the offices connected.
This solution minimizes total cost, while still connecting
all offices. There is always a path from A to B. This is
also called a minimum spanning tree.
Dijksta only works for distance graphs.
It minimizes the sum of the edges.
There can be multiple solutions with the same sum of weights.
What is a Tree:
A graph that is under tension. There are 5 definitions, but they are all the same:
-​ A graph has no cycle, and any new edge creates one cycle.
Not every tree has a root.
Binary Search Tree
Search time in O(log(N)).
Kruskal Algorithm: how do we find a
minimun spanning tree?​
The beauty of the algorithm, like Dijkstra,
is that it is a greedy algorithm. At each
step a minimum is selected, and by doing
this in steps, in the end you are going to
find the global minimum.
What you prove is that in the end you obtain an object that is a tree. So it contains all of the
nodes of the original graph, and with the edges it has it form a minimum spanning tree.
O(E*log(E)) + O(E) * (O(??) + O(1))
It all depends on how fast we can assert whether adding an edge will create a cycle.
Disjoint Set:
Start: each node is at its own disjoint set.
FInd: identify which tree is a node.
The algorithm is done when there is only 1
set.
Key Point: graph problem that is known.
There is an algorithm that is able to solve it.
Lecture 12: Exam Preparation
TIme complexity:
Guess & Check: finding a polynomial.
First check the for loop(s), it runs the same amount of times as there are guesses O(N). The
code in the for-loop computes the value of the polynomial O(1).
Best case: it is the first thing you check O(1).
Worst case: it is the last one O(N).
Average case: it is in the middle O(N/2) → O(N).
Guess & Check v2:
Average case: O((N-1)/2) → O(N).
Bisection:​
Average case: O(Log(N))
Dense Matrix Muliplication:
There are 3 for-loops which run N times.
O(N**3)