Handout 1 The University of the Witwatersrand School of Computer Science Data Structures and Algorithms: COMS2002 Introduction to Computational Problem-Solving and Critical Thinking Ritesh Ajoodha July 23, 2017 What are algorithms? What are the advantages of studying algorithms? What are some applications of algorithms? Why do we care about algorithm efficiency and data structures? In this handout, we will answer these questions. Algorithms Any unambiguous procedure that considers input values to produce output values can be considered an algorithm. Algorithms are particularly implemented to solve well-defined computational and real-world problems. The statement of the problem specifies in general terms the desired input/output relationship. The algorithm describes a specific computational procedure for achieving that input/output relationship1 . Algorithms can be used to solve difficult mathematical puzzles like the rubik cube. In fact, there exist algorithms to perform complex strong artificial intelligence to simple numerical sorting procedures. Figure 1: Rubic cubes can be solved by an algorithm. 1 Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, Clifford Stein, et al. Introduction to algorithms, volume 2. MIT press Cambridge, 2001 Introducing numerical sorting procedures forms the foundation for intermediate design techniques; data structures; and complexity analysis that will appear later in this course. Therefore, let us formally define a sorting problem: Definition 1 Suppose we need to sort a sequence of numbers: ( x1 , x2 , x3 , . . . , xn ). Then we should be able to reorder these numbers from smallest to largest: ( x10 , x20 , x30 , . . . , xn0 ), such that x10 ≤ x20 ≤ x30 ≤ . . . ≤ xn0 . Since most programming procedures require sorting as an intermediate step, sorting objects is considered as very important and so there have been several significant efforts that have improved sorting procedures over the years. However, the effectiveness of the sorting procedure utilised depends on its best and worst case complexity on the desired application. For example, we may want to consider the following factors about the data to be sorted before selecting an appropriate sorting algorithm: • How many items are to be sorted? As an example, suppose we are given the following array (list) of numbers: < 25; 65; 45; 78; 32; 19; 05; 97 >, then we should be able to sort these numbers and produce the array: < 05; 19; 25; 32; 45; 65; 78; 97 >. introduction to computational problem-solving and critical thinking 2 • Are the items already sorted? or partly sorted? c INSERTION SORT SORTING ALGORITHM • Are the items in some well-defined range or domain? • What is the architecture of the system which will perform the sorting operations? • How much or what type of storage do we have on the system? The correctness of an algorithm depends on the its ability to solve a problem with a minimal error rate - i.e. without too many incorrect outputs. If the algorithm solves the problem with no incorrect outputs, the algorithm can be said to solve the problem fully. Other algorithms which only produce some correct outputs are also possible solutions to the problem with specified error rates. Although in some cases these algorithms with error rates are useful, we are always expected to minimise the error rate for more correct solutions. In this course we will be mostly dealing with algorithms with 0 error rates, i.e. algorithms which solve given problems fully2 . What kinds of problems are solved by algorithms? Insertion sort is one of the fastest algorithms for sorting very small arrays. Complexity Memory Usage Recursion In-place Stability Comparison Sort Sorting Method Adaptability O(n2 ) O(n) No Yes+O(1) ES Yes Yes Insertion Yes Figure 2: A sorting algorithm picture card Any algorithms presented by students with a error rate > 0 will be marked incorrect. 2 Sorting objects is not the only computational problem that can be solved by using algorithms. Below are some examples of applications developed using algorithms by Computer Science Honours students at Wits University in 2013: Content-free Grammars, Gaussian Distributions, and Evolutionary Algorithms in Algorithmic Music Composition3 by Ritesh Ajoodha 3 Ritesh Ajoodha, Richard Klein, and Marija Jakovljevic. Using statistical models and evolutionary algorithms in algorithmic music composition. 2014 There have been several attempts to conduct algorithmic music composition by using techniques which resemble statistical models and evolutionary algorithms. There has, however, been no attempt to integrate these techniques into a suitable algorithm for progress in music evolution and structure. The aim of this research is to present an evolutionary approach to algorithmic music composition through machine learning. The research uses genetic algorithms to refine a statistical sample which will be formulated through Gaussian Distribution. Let it be clear that this research only poses procedural traits to develop music and does not attempt to algorithmically mimic emotional traits which are found in human-agents. The results showed that some melodies produced by the algorithm displayed significant interval changes and stylistic sense. The two-phased, Statistical and Evolutionary, model strengthens algorithmic music composition methods, particularly in the genetic and statistic phases where the key melody is defined. The potential for this research can be applied to a wider context of research in education and human-agent composition. “ ” Figure 3: Some pieces of music generated by Ajoodha et.al . 5 introduction to computational problem-solving and critical thinking Augmenting Chess Evaluation Functions with Artificial Neural Networks4 by Steven James Over the past 50 years, much work has been done in the field of computer chess to the extent that humans are no longer able to compete against the best chess programs. The large majority of this research is focused on expanding the game tree. Through a combination of improved hardware and search techniques, chess programs of today are able to analyse millions of positions per second. Despite the strength of these programs, many bemoan the fact that chess programs display no real intelligence they are simply efficient searching machines. Furthermore, top programs do not possess the ability for self-improvement. Artificial neural networks (ANNs) are one paradigm for creating self-improving agents. This paper proposes constructing a pseudo-intelligent chess program that implements an ANN. The program will be evolved using a genetic algorithm through a series of tournaments, with the final result being compared with a traditional, non-learning chess program. 3 4 James Steven. Augmenting chess evaluation functions with artificial neural networks. 2013 “ Creating A Niche Search Engine: The Crawler Component5 Figure 4: A chess scenario studied in James. ” 5 Affuta Johnathan. Creating a niche search engine: The crawler component. 2013 by Johnathan Affuta The following document is a research proposal for a research project in which we attempt to create a niche search engine, with a narrow but specific domain than is predefined by a user. Our goal is that this niche search engine will return the links of web pages that are more relevant to the user in the specified domain than other commercial search engines. This research project consists of 3 main components. These components are an focused crawler, a support vector machine (SVM) and an indexer. In order to collect the links to relevant documents the focused crawler will need to crawl the web and retrieve web documents and send them to the SVM which will give them a relevancy score. Only the documents that have a good score will be indexed. This proposal focuses mainly on the focused crawler. “ ” Rapidly Prototyping Connected Devices that Communicate over the 3G network, for the Internet of Things6 by Matthew Holmes Figure 5: Wolfram Alpha, a niche search engine. 6 Matthew Holmes. Rapidly prototyping connected devices that communicate over the 3g network, for the internet of things. 2013 The Internet of Things (IoT) is the latest buzzword amongst technology experts. A world where machines can communicate and interact without human interference is what is envisioned. Rapid prototyping toolkits have helped researchers, engineers, computer scientists, designers and enthusiasts to quickly build and test these Machine-to-Machine(M2M) systems. As the IoT begins to take form, a shift to new and exciting applications can be created “ Figure 6: The internet of things (IoT). introduction to computational problem-solving and critical thinking 4 as the 2G network is phased out and 3G and LTE become more prevalent. It is relatively simple to build a system that relies on the 2G network for communication, using these toolkits. This case study looks for a fast and simple way to rapidly prototype an IoT application with 3G connectivity. It can give confidence to South Africans who would like to rapidly prototype an application that uses 3G, but due to the limited options available, can’t be certain a given method will work. There is a risk involved in importing a 3G module as there are no recorded experimental projects that have been done in South Africa. Using smart phones when rapidly prototyping a system may not always be ideal for a non-technical user; they can add some complexity such as with the Android ADK. ” In this course you will learn about several data structures. A data structure is a way to store and organize data in order to facilitate access and modifications7 . There does not exist a single data structure that can be efficiently used to solve every problem and so clever use of data structures, with a master knowledge of their strengths and weaknesses, are needed to solve most problems. Course Objective Although you will learn many algorithms in the duration of this course, you might someday confront a problem with a solution not explicitly mentioned in this course, or any other published work on or off the Internet. Therefore, this course will teach you techniques to architect and critically examine algorithms. It is my hope that with a thorough understanding of these concepts - along with a few years of experience - you will eventually demonstrate a wide and superior palette of computational problem solving abilities. NP Problems Imagine we lived in a world where computers were infinitely fast and memory was unlimited. In this world every correct solution to a problem would work just as well as any other correct solution. Therefore, there will be no need to measure the efficiency of a problem since any solution will do. Unfortunately, while computers manipulate data faster than humanly possible, the computer’s speed is not infinite. Furthermore, while memory gets cheaper every year, it is certainly not unlimited and free. Therefore, given time and space as resources, we need to build efficient algorithms that solve our problems fully and use these resources cost-effectively. Every algorithm in this course will be critically evaluated by the amount 7 Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, Clifford Stein, et al. Introduction to algorithms, volume 2. MIT press Cambridge, 2001 N LINKED LIST 5 DATA STRUCTURE Data Structures A linked list is a group of nodes which together represent a sequence. Pros: Dynamic; Easy Insertion and Deletion; Implements other data structures. Cons: Pointers need extra storage; Nodes Stored Incontiguously = Bad Access Time; Single LL are hard to traverse; Double LL use extra storage. Figure 7: A data-structure picture card introduction to computational problem-solving and critical thinking As an example of an NP-complete problem, consider a florist with a delivery service. Every morning, it loads up each delivery truck and sends it around to deliver flowers to several addresses. At the end of the day, each truck must end up back at the florist so that it is ready to be loaded for the next day. To reduce costs, the company wants to select an order of delivery stops that yields the lowest overall distance traveled by each truck. This problem is the well-known "traveling-salesman problem", and it is NP-complete. It has no known efficient solution. Under certain assumptions, however, we know of efficient algorithms that give an overall distance which is not too far above the smallest possible. L THE TRAVELLING SALESMAN ∞ NP PROBLEM of time it takes to run. That is, given a respective input how long will the algorithm take to produce the desired output? This is an efficiency measure using speed. However, there are some problems with no known efficient solution, known as NP-complete problems. Although no efficient solution has ever been presented, there has also been no proof that suggests that a solution does not exists. Furthermore, this intriguing subset of problems has a property that if an efficient solution can be found for one of them, then an efficient solution exists for all of them. Lastly, many NP-complete problems are similar to problems that have efficient solutions. It is interesting how slight alterations to the problem statement can cause a tremendous change in computational efficiency. It is important to understand NP-completeness since these problems can arise in real applications, and so if you can’t recognise these problems then you may waste time trying to obtain an efficient solution where none might exist. 5 Formulated in 1930, this is one of the most intensively studied problems in optimization. Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the origin city? NP-Hard introduction to computational problem-solving and critical thinking 6 Tutorial 1 1. Implement each of the following algorithms below using a pen and paper: Adding Two numbers Step 1: Start Step 2: Declare variables num1, num2 and sum. Step 3: Read values num1 and num2. Step 4: Add num1 and num2 and assign the result to sum: sum ← num1+num2 Step 5: Display sum Step 6: Stop Find the largest of 3 numbers: Step 1: Start Step 2: Declare variables a,b and c. Step 3: Read variables a,b and c. Step 4: If a>b If a>c Display a is the largest number. Else Display c is the largest number. Else If b>c Display b is the largest number. Else Display c is the greatest number. Step 5: Stop Find the nth ≤ 1000 term of the Fibonacci series Step 1: Start Step 2: Declare variables first_term,second_term and temp. Step 3: Initialize variables first_term←0 second_term←1 Step 4: Display first_term and second_term Step 5: Repeat the steps until second_term1000 5.1: temp←second_term 5.2: second_term←second_term+first term 5.3: first_term←temp 5.4: Display second_term Step 6: Stop Determine if a number is prime or not Step 1: Start Step 2: Declare variables n,i,flag. Step 3: Initialize variables flag←1 i←2 Step 4: Read n from user. Step 5: Repeat the steps until i<(n/2) 5.1 If remainder of n÷i equals 0 flag←0 Go to step 6 5.2 i←i+1 Step 6: If flag=0 Display n is not prime else Display n is prime Step 7: Stop 2. Provide some examples of algorithmic procedures (e.g. How Google search engine works; learning the lyrics to a song; applying for a VISA to go to Germany; or simply baking a cake. Describe all of the processes involved in the system as well as how these processes are dependent on the completion of other processes. 3. Suppose we are comparing implementations of of two sorting algorithms: Sorting Algorithm 1 and Sorting Algorithm 2 on the same machine. For inputs of size n, Sorting Algorithm 1 runs in 8n2 steps, while Sorting Algorithm 2 introduction to computational problem-solving and critical thinking 7 runs in 64nlgn steps. For which values of n does Sorting Algorithm 1 beat Sorting Algorithm 2? 4. What is the smallest value of n such that Searching Algorithm 1’s running time is 100n2 runs faster than Searching Algorithm 2’s running time is 2n on the same machine? 5. Come up with a real-world problem in which only the best solution will do. Then come up with one in which a solution that is approximately the best is good enough. 6. There are two algorithms given below: The Travelling Salesman problem and Dijkstra’s Algorithm. Read the picture cards below and answer the following question: How are Dijkstra’s Algorithm and traveling-salesman problem similar? How are they different? ∞ Formulated in 1930, this is one of the most intensively studied problems in optimization. Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the origin city? NP-Hard t POPULAR ALGORITHM THE TRAVELLING SALESMAN NP PROBLEM L DIJKSTRA’S ALGORITHM 75 Dijkstra’s algorithm is used to find the shortest path bewteen two nodes in a graph. Worst Case: O(| E| + |V |log|V |) Dijkstra designed the shortest path algorithm in about 20 minutes without aid of paper and pen and later implemented it for ARMAC for a slightly simplified transportation map of 64 cities in Netherland. 7. Select a data structure that you have seen previously for example: linked lists; arrays; trees; graphs; stacks; or queues, and discuss it’s strengths and weaknesses. 8. Other than speed, what other measures of efficiency might one use in a real-world setting? Laboratory Questions 1. Write a program that prints ‘Hello World’ to the screen. 2. Write a program that asks the user for his name and greets him with his name. 3. Modify the previous program such that only the users Alice and Bob are greeted with their names. 4. Write a program that asks the user for a number n and prints the sum of the numbers 1 to n 5. Modify the previous program such that only multiples of three or five are considered in the sum, e.g. 3, 5, 6, 9, 10, 12, 15 for n=17 6. Write a program that asks the user for a number n and gives him the possibility to choose between computing the sum and computing the product of 1,...,n. introduction to computational problem-solving and critical thinking 8 7. Write a program that prints a multiplication table for numbers up to 12. 8. Write a program that prints all prime numbers. (Note: if your programming language does not support arbitrary size numbers, printing all primes up to the largest number you can represent is fine too.) 9. Write a guessing game where the user has to guess a secret number. After every guess the program tells the user whether his number was too large or too small. At the end the number of tries needed should be printed. I counts only as one try if the user inputs the same number consecutively. 10. Write a program that prints the next 20 leap years. 11. Write a program that computes: 4. ∑ k = 1106(−1)k + 12k − 1 = 4.(1 − 1 1 1 1 1 + − + − ...) 3 5 7 9 11 (1) 12. Write a function that returns the largest element in a list. 13. Write function that reverses a list, preferably in place. 14. Write a function that checks whether an element occurs in a list. 15. Write a function that returns the elements on odd positions in a list. 16. Write a function that computes the running total of a list. 17. Write a function that tests whether a string is a palindrome. 18. Write three functions that compute the sum of the numbers in a list: using a for-loop, a while-loop and recursion. 19. Write a function on_all that applies a function to every element of a list. Use it to print the first twenty perfect squares. 20. Write a function that concatenates two lists. 21. Write a function that combines two lists by alternatively taking elements, e. g. [a,b,c], [1,2,3] → [a,1,b,2,c,3]. 22. Write a function that merges two sorted lists into a new list. 23. Write a function that computes the list of the first 100 Fibonacci numbers. 24. Write a function that takes a number and returns a list of its digits. 25. Write functions that add, subtract, and multiply two numbers in their digit-list representation (and return a new digit list). If you’re ambitious you can implement Karatsuba multiplication. Try different bases. What is the best base if you care about speed? 26. Implement the following sorting algorithms: Selection sort, Insertion sort, Merge sort, Quick sort, Stooge Sort. Check Wikipedia for descriptions. 27. Implement binary search. 28. Write a function that takes a list of strings an prints them, one per line, in a rectangular frame. For example the list ["Hello", "World", "in", "a", "frame"] gets printed as: ********* * Hello * * World * * in * * a * * frame * ********* introduction to computational problem-solving and critical thinking 9 29. Write function that translated a text to Pig Latin and back. English is translated to Pig Latin by taking the first letter of every word, moving it to the end of the word and adding ‘ay’. “The quick brown fox” becomes “Hetay uickqay rownbay oxfay”. 30. Implement an algorithm to determine if a string has all unique characters. What if you cannot use additional data structures? 31. Implement a function void reverse(char* str) in C or C++ which reverses a nullterminated string. 32. Given two strings, write a method to decide if one is a permutation of the other. 33. Write a method to replace all spaces in a string with’%20’. You may assume that the string has sufficient space at the end of the string to hold the additional characters, and that you are given the "true" length of the string. EXAMPLE Input: "Mr John Smith Output: "Mr%20Dohn%20Smith” 34. Implement a method to perform basic string compression using the counts of repeated characters. For example, the string aabcccccaaa would become a2blc5a3. If the "compressed" string would not become smaller than the original string, your method should return the original string. 35. Given an image represented by an NxN matrix, where each pixel in the image is 4 bytes, write a method to rotate the image by 90 degrees. Can you do this in place? 36. Write an algorithm such that if an element in an MxN matrix is 0, its entire row and column are set to 0. 37. Assume you have a method isSubstring which checks if one word is a substring of another. Given two strings, si and s2, write code to check if s2 is a rotation of si using only one call to isSubstring (e.g.,"waterbottle"is a rotation of "erbottlewat"). References Ritesh Ajoodha, Richard Klein, and Marija Jakovljevic. Using statistical models and evolutionary algorithms in algorithmic music composition. 2014. Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, Clifford Stein, et al. Introduction to algorithms, volume 2. MIT press Cambridge, 2001. Matthew Holmes. Rapidly prototyping connected devices that communicate over the 3g network, for the internet of things. 2013. Affuta Johnathan. Creating a niche search engine: The crawler component. 2013. James Steven. Augmenting chess evaluation functions with artificial neural networks. 2013.