Introduction to Python Instructor : Rupal Mishra Indian Institute of Quantitative Finance Introduction to Python Python is a high-level programming language ⚫ Open source and community driven ⚫ standard distribution includes many modules ⚫ Dynamic typed ⚫ Source can be compiled or run just-in-time ⚫ Similar to perl, tcl, ruby ⚫ 2 Why Python? Unlike AML and Avenue, there is a considerable base of developers already using the language ⚫ “Tried and true” language that has been in development since 1991 ⚫ Can interface with the Component Object Model (COM) used by Windows ⚫ Can interface with Open Source GIS toolsets ⚫ 3 Python Interfaces IDLE – a cross-platform Python development environment ⚫ PythonWin – a Windows only interface to Python ⚫ Python Shell – running 'python' from the Command Line opens this interactive shell ⚫ For the exercises, we'll use IDLE, but you can try them all and pick a favorite ⚫ 4 IDLE – Development Environment ⚫ IDLE helps you program in Python by: – color-coding your program code – debugging – auto-indent – interactive shell 5 Example Python ⚫ Hello World print “hello world” Prints hello world to standard out ⚫ Open IDLE and try it out yourself ⚫ Follow along using IDLE ⚫ 6 More than just printing Python is an object oriented language ⚫ Practically everything can be treated as an object ⚫ “hello world” is a string ⚫ Strings, as objects, have methods that return the result of a function on the string ⚫ 7 String Methods Assign a string to a variable ⚫ In this case “hw” ⚫ hw.title() ⚫ hw.upper() ⚫ hw.isdigit() ⚫ hw.islower() ⚫ 8 String Methods The string held in your variable remains the same ⚫ The method returns an altered string ⚫ Changing the variable requires reassignment ⚫ – hw = hw.upper() – hw now equals “HELLO WORLD” 9 Keywords and identifiers Cannot use keyword as variable ⚫ Keywords are case sensitive ⚫ In python 3.9, there were 36 keywords ⚫ All keywords except for True ,False and None are in lowercase. ⚫ Identifier – Any combination of letter in lowercase or uppercase or in digits or an underscore _. Like ‘variable_2’ correct identifier. ⚫ Identifier cannot start with a digit.1x is not valid identifier. Indian Institute of Quantitative Finance ⚫ Coding Statements Assignment is a coding statement like a=1 ⚫ Multi-line assignment – use this (\) and then go to next line. ⚫ Multiline can be achieved using (),[] or {}. ⚫ For indentation, use (: ) ⚫ Indian Institute of Quantitative Finance Python Lists: Lists are the most versatile of Python's compound data types. A list contains items separated by commas and enclosed within square brackets ([]). ⚫ To some extent, lists are similar to arrays in C. One difference between them is that all the items belonging to a list can be of different data type. ⚫ Python Lists: The values stored in a list can be accessed using the slice operator ( [ ] and [ : ] ) with indexes starting at 0 in the beginning of the list and working their way to end-1. ⚫ The plus ( + ) sign is the list concatenation operator, and the asterisk ( * ) is the repetition operator. ⚫ Indian Institute of Quantitative Finance Lists Think of a list as a stack of cards, on which your information is written ⚫ The information stays in the order you place it in until you modify that order ⚫ Methods return a string or subset of the list or modify the list to add or remove components ⚫ Written as var[index], index refers to order within set (think card number, starting at 0) ⚫ You can step through lists as part of a loop ⚫ 14 Flow of code to print alternate elements in list: ⚫ Loop : for each element in list, if position/2 !=0 , print that element. ⚫ Indian Institute of Quantitative Finance Lists list = ['iiqf', 243 , 3.9, 'Don', 28.9 ] tinylist = [123, 'Don'] print(list) # print(list[0]) # print(list[1:3]) # 3rd print(list[2:]) # element print(tinylist * 2) # print(list + tinylist) Prints complete list Prints first element of the list Prints elements starting from 2nd till Prints elements starting from 3rd Prints list two times # Prints concatenated lists Output: ['iiqf', 243, 3.9, 'Don', 28.9] iiqf [243, 3.9] [3.9, 'Don', 28.9] [123, 'Don', 123, 'Don'] ['iiqf', 243, 3.9, 'Don', 28.9, 123, 'Don'] List Methods ⚫ Adding to the List – var[n] = object ⚫ replaces n with object – var.append(object) ⚫ ⚫ adds object to the end of the list Removing from the List – var[n] = [] ⚫ empties contents of card, but preserves order – var.remove(n) ⚫ removes card at n – var.pop(n) ⚫ removes n and returns its value 17 List Methods list[1] = 246 print(list) list.append(‘xyz’) print(list) list[3]=[] print(list) list.remove(28.9) print(list) list.pop(0) print(list) Indian Institute of Quantitative Finance Python Tuples: A tuple is another sequence data type that is similar to the list. A tuple consists of a number of values separated by commas. Unlike lists, however, tuples are enclosed within parentheses. ⚫ The main differences between lists and tuples are: Lists are enclosed in brackets ( [ ] ), and their elements and size can be changed, while tuples are enclosed in parentheses ( ( ) ) and cannot be updated. Tuples can be thought of as read-only lists. ⚫ Tuple tuple = ( 'iiqf', 243 , 2.23, 'sanjay', 70.2 ) tinytuple = (123, 'singhania') print(tuple) # Prints complete list print(tuple[0]) # Prints first element of the list print(tuple[1:3]) # Prints elements starting from 2nd till 3rd print(tuple[2:]) # Prints elements starting from 3rd element print(tinytuple * 2) # Prints list(two times print(tuple + tinytuple) # Prints concatenated lists Output: List vs Tuple Lists Tuple Lists are mutable like list. append will work. Tuples are immutable. There is no method append here. Iterations in list are slow Iterations are comparatively faster List is better for addition and removal of elements Tuple is fast, so better for accessing elements. Consumes more memory Consumes less memory Multiple methods in list Less methods in list. More prone to errors Less prone to errors Indian Institute of Quantitative Finance Python Dictionary: Python 's dictionaries are hash table type. They work like associative arrays or hashes found in Perl and consist of key-value pairs. ⚫ Keys can be almost any Python type, but are usually numbers or strings. Values, on the other hand, can be any arbitrary Python object. ⚫ Dictionaries are enclosed by curly braces ( { } ) and values can be assigned and accessed using square braces ( [] ). ⚫ Dictionaries Dictionaries are sets of key & value pairs ⚫ Allows you to identify values by a descriptive name instead of order in a list ⚫ Keys are unordered unless explicitly sorted ⚫ Keys should be immutable. Values can be mutable or immutable. ⚫ Keys are unique: ⚫ – var[‘item’] = “apple” – var[‘item’] = “banana” – Print(var[‘item’]) prints just banana 23 Data Type Conversion: Function int(x [,base]) Description Converts x to an integer. base specifies the base if x is a string. long(x [,base] ) Converts x to a long integer. base specifies the base if x is a string. float(x) Converts x to a floating-point number. complex(real [,imag]) str(x) Creates a complex number. repr(x) Converts object x to an expression string. eval(str) Evaluates a string and returns an object. tuple(s) Converts s to a tuple. list(s) Converts s to a list. set(s) Converts s to a set. dict(d) Creates a dictionary. d must be a sequence of (key,value) tuples. frozenset(s) Converts s to a frozen set. chr(x) Converts an integer to a character. unichr(x) Converts an integer to a Unicode character. ord(x) Converts a single character to its integer value. hex(x) Converts an integer to a hexadecimal string. oct(x) Converts an integer to an octal string. Converts object x to a string representation. 4. Python - Basic Operators Python language supports following type of operators. ⚫ Arithmetic Operators ⚫ Comparision Operators ⚫ Logical (or Relational) Operators ⚫ Assignment Operators ⚫ Conditional (or ternary) Operators ⚫ Python Arithmetic Operators: Operator + * / % ** // Description Addition - Adds values on either side of the operator Subtraction - Subtracts right hand operand from left hand operand Multiplication - Multiplies values on either side of the operator Division - Divides left hand operand by right hand operand Modulus - Divides left hand operand by right hand operand and returns remainder Exponent - Performs exponential (power) calculation on operators Floor Division - The division of operands where the result is the quotient in which the digits after the decimal point are removed. Example a + b will give 30 a - b will give -10 a * b will give 200 b / a will give 2 b % a will give 0 a**b will give 10 to the power 20 9//2 is equal to 4 and 9.0//2.0 is equal to 4.0 Python Comparison Operators: Operato Description r == Checks if the value of two operands are equal or not, if yes then condition becomes true. != Checks if the value of two operands are equal or not, if values are not equal then condition becomes true. <> Checks if the value of two operands are equal or not, if values are not equal then condition becomes true. > Checks if the value of left operand is greater than the value of right operand, if yes then condition becomes true. < Checks if the value of left operand is less than the value of right operand, if yes then condition becomes true. >= Checks if the value of left operand is greater than or equal to the value of right operand, if yes then condition becomes true. <= Checks if the value of left operand is less than or equal to the value of right operand, if yes then condition becomes true. Example (a == b) is not true. (a != b) is true. (a <> b) is true. This is similar to != operator. (a > b) is not true. (a < b) is true. (a >= b) is not true. (a <= b) is true. Python Assignment Operators: Operator = += -= *= /= %= **= //= Description Simple assignment operator, Assigns values from right side operands to left side operand Example c = a + b will assigne value of a + b into c Add AND assignment operator, It adds right operand to c += a is equivalent the left operand and assign the result to left operand to c = c + a Subtract AND assignment operator, It subtracts right c -= a is equivalent operand from the left operand and assign the result to left to c = c - a operand Multiply AND assignment operator, It multiplies right c *= a is equivalent operand with the left operand and assign the result to left to c = c * a operand Divide AND assignment operator, It divides left operand c /= a is equivalent with the right operand and assign the result to left to c = c / a operand Modulus AND assignment operator, It takes modulus c %= a is equivalent using two operands and assign the result to left operand to c = c % a Exponent AND assignment operator, Performs exponential c **= a is (power) calculation on operators and assign value to the equivalent to c = c left operand ** a Floor Division and assigns a value, Performs floor division c //= a is equivalent on operators and assign value to the left operand to c = c // a Python Bitwise Operators: Operat Description or & Binary AND Operator copies a bit to the result if it exists in both operands. | Binary OR Operator copies a bit if it exists in either operand. ^ Binary XOR Operator copies the bit if it is set in one operand but not both. ~ Binary Ones Complement Operator is unary and has the effect of 'flipping' bits. << Binary Left Shift Operator. The left operands value is moved left by the number of bits specified by the right operand. >> Binary Right Shift Operator. The left operands value is moved right by the number of bits specified by the right operand. Example (a & b) will give 12 which is 0000 1100 (a | b) will give 61 which is 0011 1101 (a ^ b) will give 49 which is 0011 0001 (~a ) will give -60 which is 1100 0011 a << 2 will give 240 which is 1111 0000 a >> 2 will give 15 which is 0000 1111 Python Logical Operators: Operat Description or and Called Logical AND operator. If both the operands are true then then condition becomes true. or Called Logical OR Operator. If any of the two operands are non zero then then condition becomes true. not Called Logical NOT Operator. Use to reverses the logical state of its operand. If a condition is true then Logical NOT operator will make false. Example (a and b) is true. (a or b) is true. not(a and b) is false. Python Membership Operators: ⚫ In addition to the operators discussed previously, Python has membership operators, which test for membership in a sequence, such as strings, lists, or tuples. Operator Description Example in Evaluates to true if it finds a variable in the x in y, here in results in a specified sequence and false otherwise. 1 if x is a member of sequence y. not in Evaluates to true if it does not finds a x not in y, here not in variable in the specified sequence and false results in a 1 if x is a otherwise. member of sequence y. Python Operators Precedence Operator Description ** Exponentiation (raise to the power) ~+* / % // Ccomplement, unary plus and minus (method names for the last two are +@ and -@) Multiply, divide, modulo and floor division +- Addition and subtraction >> << Right and left bitwise shift & Bitwise 'AND' ^| Bitwise exclusive `OR' and regular `OR' <= < > >= Comparison operators <> == != Equality operators = %= /= //= -= += Assignment operators *= **= is is not Identity operators in not in Membership operators not or and Logical operators Python - IF...ELIF...ELSE Statement The syntax of the if statement is: if expression: if expression: statement(s) statement(s) else: Example: statement(s) var1 = 100 if var1: print ("1 - Got a true expression value“) print (var1) var2 = 0 if var2: print ("2 - Got a true expression value“) print (var2) ⚫ var1 = 100 if var1: print("1 - Got a true expression value") print(var1) else: print("1 - Got a false expression value") print(var1) var2 = 0 if var2: print("2 - Got a true expression value") print(var2) else: print("2 - Got a false expression value") print(var2) The Nested if...elif...else Construct var = 100 if var < 200: print("Expression value is less than 200") if var == 150: print("Which is 150") elif var == 100: print("Which is 100") elif var == 50: print("Which is 50") elif var < 50: print("Expression value is less than 50") else: print("Could not find true expression") Single Statement Suites: If the suite of an if clause consists only of a single line, it may go on the same line as the header statement: x=1 if (x==1): print("x is 1") 5. Python - while Loop Statements The while loop is one of the looping constructs available in Python. The while loop continues until the expression becomes false. The expression has to be a logical expression and must return either a true or a false value The syntax of the while loop is: while expression: statement(s) Example: x = 0 while (x < 9): print ('The count is:', x) x = x + 1 The Infinite Loops: You must use caution when using while loops because of the possibility that this condition never resolves to a false value. This results in a loop that never ends. Such a loop is called an infinite loop. An infinite loop might be useful in client/server programming where the server needs to run continuously so that client programs can communicate with it as and when required. Following loop will continue till you enter CTRL+C : FLIP OF COIN GAME: Either you win or loose at flip of coin. Single Statement Suites: ⚫ Similar to the if statement syntax, if your while clause consists only of a single statement, it may be placed on the same line as the while header. ⚫ Here is the syntax of a one-line while clause: while expression : statement 6. Python - for Loop Statements The for loop in Python has the ability to iterate over the items of any sequence, such as a list or a string. The syntax of the loop look is: for iterating_var in sequence: statements(s) Example: str = "Python" for i in str: print(i) Python break,continue and pass Statements The break Statement: ⚫ The break statement in Python terminates the current loop and resumes execution at the next statement, just like the traditional break found in C. Example: for letter in 'IIQFNew': if(letter == 'N'): break print('Current Letter :', letter) var = 10 while var>0: print('Current var value :', var) var = var-1 if var == 5: break The continue Statement: ⚫ The continue statement in Python returns the control to the beginning of the while loop. The continue statement rejects all the remaining statements in the current iteration of the loop and moves the control back to the top of the loop. Example: for letter in 'IIQFNew': if letter == 'N': continue print('Current Letter :', letter) var = 10 while var > 0: var = var -1 if var == 5: continue print('Current variable value :', var) The else Statement Used with Loops Python supports to have an else statement associated with a loop statements. ⚫ If the else statement is used with a for loop, the else statement is executed when the loop has exhausted iterating the list. ⚫ If the else statement is used with a while loop, the else statement is executed when the condition becomes false. Example: for num in range(10,20): for i in range(2,num): if num%i == 0: j=num/i print('%d equals %d * %d' % (num,i,j)) break else: print(num, 'is a prime number') The pass Statement: The pass statement in Python is used when a statement is required syntactically but you do not want any command or code to execute. ⚫ The pass statement is a null operation; nothing happens when it executes. The pass is also useful in places where your code will eventually go, but has not been written yet (e.g., in stubs for example): Example: ⚫ for letter in 'Python': if letter == 'h': pass print('This is pass block') print('Current Letter :', letter) What is Numpy? Numpy, Scipy, and Matplotlib provide MATLAB-like functionality in python. ⚫ Numpy Features: ⚫ – Typed multidimentional arrays (matrices) – Fast numerical computations (matrix math) – High-level math functions 45 Why do we need NumPy Python does numerical computations slowly. ⚫ 1000 x 1000 matrix multiply ⚫ – Python triple loop takes > 10 min. – Numpy takes ~0.03 seconds 46 NumPy Overview 1. 2. 3. 4. Arrays Shaping and transposition Mathematical Operations Indexing and slicing 47 Arrays Structured lists of numbers. Vectors ⚫ Matrices ⚫ Images ⚫ Tensors ⚫ ConvNets ⚫ 48 Arrays Structured lists of numbers. ⚫ Vectors ⚫ Matrices ⚫ Images ⚫ Tensors ⚫ ConvNets ⚫ 49 Arrays Structured lists of numbers. Vectors ⚫ Matrices ⚫ Images ⚫ Tensors ⚫ ConvNets ⚫ 50 Arrays Structured lists of numbers. Vectors ⚫ Matrices ⚫ Images ⚫ Tensors ⚫ ConvNets ⚫ 51 Arrays Structured lists of numbers. Vectors ⚫ Matrices ⚫ Images ⚫ Tensors ⚫ ConvNets ⚫ 52 Arrays, Basic Properties import numpy as np a = np.array([[1,2,3],[4,5,6]],dtype=np.float32) print a.ndim, a.shape, a.dtype 1. 2. 3. Arrays can have any number of dimensions, including zero (a scalar). Arrays are typed: np.uint8, np.int64, np.float32, np.float64 Arrays are dense. Each element of the array exists and has the same type. 53 Arrays, creation np.ones, np.zeros ⚫ np.arange ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.random ⚫ 54 Arrays, creation np.ones, np.zeros ⚫ np.arange ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.random ⚫ 55 Arrays, creation np.ones, np.zeros ⚫ np.arrange arange([start,]stop[,step],[,dtype=None] ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.random ⚫ 56 Arrays, creation np.ones, np.zeros ⚫ np.arange ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.random ⚫ 57 Axis in Numpy Indian Institute of Quantitative Finance np.ones, np.zeros ⚫ np.arange ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.random ⚫ 59 Arrays, creation np.ones, np.zeros ⚫ np.arange ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.random ⚫ 60 Arrays, creation np.ones, np.zeros ⚫ np.arange ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.random ⚫ 61 Arrays, creation np.ones, np.zeros ⚫ np.arange ⚫ np.concatenate ⚫ np.astype ⚫ np.zeros_like, np.ones_like ⚫ np.random.rando m ⚫ 62 Arrays, danger zone Must be dense, no holes. ⚫ Must be one type ⚫ Cannot combine arrays of different shape ⚫ 63 Shaping a = np.array([1,2,3,4,5,6]) a = a.reshape(3,2) a = a.reshape(2,-1) a = a.ravel() 1. Total number of elements cannot change. 2. Use -1 to infer axis shape 3. Row-major by default (MATLAB is column-major) 64 Transposition a = np.arange(10).reshape(5,2) a = a.T a = a.transpose((1,0)) np.transpose permutes axes. a.T transposes the first two axes. 65 Mathematical operators Arithmetic operations are element-wise ⚫ Logical operator return a bool array ⚫ In place operations modify the array ⚫ 66 Mathematical operators Arithmetic operations are element-wise ⚫ Logical operator return a bool array ⚫ In place operations modify the array ⚫ 67 Mathematical operators Arithmetic operations are element-wise ⚫ Logical operator return a bool array ⚫ In place operations modify the array ⚫ 68 Mathematical operators Arithmetic operations are element-wise ⚫ Logical operator return a bool array ⚫ In place operations modify the array ⚫ 69 Math, universal functions Also called ufuncs Element-wise Examples: – – – – – np.exp np.sqrt np.sin np.cos np.isnan 70 Quick Game ⚫ Create a game, where you guess a number between 1 to 10 and check it with random integer number generated by the code. If number matches, you win else loose. Indian Institute of Quantitative Finance Indexing x[0,0] # top-left element x[0,-1] # first row, last column x[0,:]# first row (many entries) x[:,0]# first column (many entries) Notes: – Zero-indexing – Multi-dimensional indices are comma-separated (i.e., a tuple) 72 Scipy ⚫ • Scipy(https://www.scipy.org/): Scientific Python ⚫ • often mentioned in the same breath with NumPy ⚫ • based on the data structures of Numpy and furthermore its basic creation and manipulation functions ⚫ • Both numpy and scipy has to be installed. Numpy has to be installed before scipy Indian Institute of Quantitative Finance Numpy vs Scipy ⚫ Numpy: • Numpy is written in C and use for mathematical or numeric calculation. • It is faster than other Python Libraries • Numpy is the most useful library for Data Science to perform basic calculations. • Numpy contains nothing but array data type which performs the most basic operation like sorting, shaping, indexing, etc. Indian Institute of Quantitative Finance Numpy vs Scipy ⚫ SciPy: • SciPy is built in top of the NumPy • SciPy is a fully-featured version of Linear Algebra while Numpy contains only a few features. • Most new Data Science features are available in Scipy rather than Numpy. Indian Institute of Quantitative Finance Statistics • The main public methods for continuous random variables are: • rvs: Random Variates 1) pdf: Probability Density Function 2) cdf: Cumulative Distribution Function 3) sf: Survival Function (1-CDF) Indian Institute of Quantitative Finance Statistics 1) ppf: Percent Point Function (Inverse of CDF) 2) isf: Inverse Survival Function (Inverse of SF) 3) stats: Return mean, variance, (Fisher’s) skew, or (Fisher’s) kurtosis moment: noncentral moments of the distribution Indian Institute of Quantitative Finance Tips to avoid bugs 1. 2. 3. 4. 5. Know what your datatypes are. Check whether you have a view or a copy. Use matplotlib for sanity checks. Use pdb to check each step of your computation. Know np.dot vs np.mult. 78 Scipy ⚫ Statistics(Scipy.Stats) : from scipy import stats import numpy as np from scipy.stats import uniform print(stats.norm.cdf(0)) print(stats.norm.cdf(0.5)) print(stats.norm.cdf(np.array([-3.33, 0, 3.33]))) print(stats.norm.ppf(0.5)) print(stats.norm.ppf(3)) Indian Institute of Quantitative Finance Pandas 1. fast and efficient Data Frame object for data manipulation 2. Tools for reading and writing data between in-memory data structures and different formats: CSV and text files, Microsoft Excel, SQL databases Indian Institute of Quantitative Finance Pandas 1. reshaping and pivoting of data sets 2. Intelligent label-based slicing, fancy indexing, and sub setting of large data sets; 3. Columns can be inserted and deleted from data structures for size mutability; Indian Institute of Quantitative Finance Exercise ⚫ Run a code for simulation using pandas and numpy using GBM model. Indian Institute of Quantitative Finance Indentation and Blocks Python uses whitespace and indents to denote blocks of code ⚫ Lines of code that begin a block end in a colon: ⚫ Lines within the code block are indented at the same level ⚫ To end a code block, remove the indentation ⚫ You'll want blocks of code that run only when certain conditions are met ⚫ 83 Modules Modules are additional pieces of code that further extend Python’s functionality ⚫ A module typically has a specific function ⚫ – additional math functions, databases, network… Python comes with many useful modules ⚫ arcgisscripting is the module we will use to load ArcGIS toolbox functions into Python ⚫ 84 Modules ⚫ Modules are accessed using import – import sys, os # imports two modules ⚫ Modules can have subsets of functions – os.path is a subset within os ⚫ Modules are then addressed by modulename.function() – sys.argv # list of arguments – filename = os.path.splitext("points.txt") – filename[1] # equals ".txt" 85