Introduction to Python
Session 2: Beginning Numerical Python and Visualization
Jeremy Chen
This Session’s Agenda
•
•
•
•
•
NumPy Arrays
Random Number Generation
Numerical Linear Algebra
Visualization
An OLS Example
2
新加坡国立大学商学院
Preliminaries
• By now you should have a Python
distribution…
– Run IPython or Spyder
– If not, start downloading one and work with
IDLE
3
新加坡国立大学商学院
NumPy Arrays
• The ndarray is a typed N-dimensional
array object:
>>> import numpy as np
>>> A = np.ndarray((2,3))
# Kinda the same as np.empty((2,3)); Uninitialized data
array([[ 8.48798326e-314,
2.12199579e-314,
0.00000000e+000],
[ 0.00000000e+000,
1.05290307e-253,
1.47310613e-319]])
>>> A[0,0] = 2; A[0,1] = 3; A[1,1] = 4; A[1,2] = 5; A
array([[ 2., 3., 0.],
[ 0., 4., 5.]])
>>> 10 * A + 1
# Simple vector operations
array([[ 21., 31.,
1.],
[ 1., 41., 51.]])
>>> np.exp(A)
# And various other vector operations
array([[
7.3890561 ,
20.08553692,
1.
],
[
1.
,
54.59815003, 148.4131591 ]])
>>> [A.shape, A.ndim]
# And don’t forget ndarrays have a "shape" and "dimension"
[(2, 3), 2]
>>> A.dtype
# ... and a data type.
dtype('float64')
4
新加坡国立大学商学院
NumPy Arrays: Creating
• Common ways of creating a ndarray
>>> np.array(np.arange(5))
# arange is like range but returns a ndarray
array([0, 1, 2, 3, 4])
>>> np.array(np.arange(5)).dtype
# numpy selects an appropriate datatype
dtype('int32')
>>> np.reshape([3,2,1,3,1,2,6,7], (2,4))
array([[3, 2, 1, 3],
[1, 2, 6, 7]])
>>> np.ones((2,3,4))
array([[[ 1., 1., 1., 1.],
[ 1., 1., 1., 1.],
[ 1., 1., 1., 1.]],
[[ 1., 1., 1., 1.],
[ 1., 1., 1., 1.],
[ 1., 1., 1., 1.]]])
>>> np.zeros((5))
array([ 0., 0., 0., 0., 0.])
5
新加坡国立大学商学院
NumPy Arrays: Creating
• An amusing way to create a ndarray
>>> M = np.mat('[1,2,3;4,5,6]'); M
# Like MATLAB
matrix([[1, 2, 3],
[4, 5, 6]])
>>> A = np.array(M); A
# ... not really necessary
array([[1, 2, 3],
[4, 5, 6]])
>>> # Here's the obligatory Matrix Multiplication digression
>>> M * M.T
# Matrix multiplication (M.T is the transpose of M)
matrix([[14, 32],
[32, 77]])
>>> A * A.T
# Not matrix multiplication
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (2,3) (3,2)
>>> A.dot(A.T)
# Matrix multiplication for 2 dimensional ndarrays
array([[14, 32],
[32, 77]])
>>> np.dot(A,A.T) # ... same here. (Dot product for 1 dimensional ndarrays)
array([[14, 32],
[32, 77]])
新加坡国立大学商学院
6
NumPy Arrays: Accessing
• Simple slicing creates views (not copies)
>>> A = np.reshape(np.arange(2*4)+1, (2,4)); A
array([[1, 2, 3, 4],
[5, 6, 7, 8]])
>>> B = A[:,1:3]; B
array([[2, 3],
[6, 7]])
>>> B[0,0] = 100; B; A
array([[100,
3],
[ 6,
7]])
array([[ 1, 100,
3,
4],
[ 5,
6,
7,
8]])
• “Fancy Indexing” creates copies
>>> C = A[:, [1,3]]; C
array([[100,
4],
[ 6,
8]])
>>> C[0,0] = 200; C; A
array([[200,
4],
[ 6,
8]])
array([[ 1, 100,
3,
[ 5,
6,
7,
新加坡国立大学商学院
# Using lists/arrays for indexing
4],
8]])
7
NumPy Arrays: A Little More
• Some standard operations
>>> A = np.reshape(np.arange(2*4)+1, (2,4)); A
array([[1, 2, 3, 4],
[5, 6, 7, 8]])
>>> A.sum()
36
>>> A.sum(axis=0)
array([ 6, 8, 10, 12])
>>> A.sum(axis=1)
array([10, 26])
>>> A.cumsum(axis=1)
array([[ 1, 3, 6, 10],
[ 5, 11, 18, 26]])
>>> (A.mean(axis=1), A.std(axis=1), A.var(axis=1), A.min(axis=1), A.argmax(axis=1))
(array([ 2.5,
6.5]), array([ 1.11803399,
1.11803399]), array([ 1.25,
1.25]), array([1, 5]), array([3, 3]))
>>> arr = np.array([1,2,4,9,1,4]); arr.sort(); arr
# Sorting
array([1, 1, 2, 4, 4, 9])
>>> np.unique(arr)
# Keep only unique entries
array([1, 2, 4, 9])
>>> np.unique(np.array(['A', 'B', 'CC', 'c', 'A', 'CC']))
# Applies to strings too
array(['A', 'B', 'CC', 'c'],
dtype='|S2')
新加坡国立大学商学院
8
Random Number Generation
• The Random Seed
>>> np.random.seed(10)
• Various Distributions
>>> np.random.SOME_DISTRIBUTION(params, size=(dimensions))
>>> # rand ~ Uniform; randn ~ Normal; exponential; beta;
>>> # gamma; binomial; randint(=>Discrete) ... even triangular
9
新加坡国立大学商学院
Numerical Linear Algebra
• Matrix Decompositions
>>> A = np.ones((5,5)) + np.eye(5)
>>> L = np.linalg.cholesky(A)
# Cholesky decomposition.
>>> np.allclose(np.dot(L,L.T), A)
# Check...
True
>>> Q, R = np.linalg.qr(A)
# QR decomposition
>>> np.allclose(np.dot(Q,R), A)
True
>>> # Eigenvalue decomposition (Use eigh when symmetric/Hermetian)
>>> e, V = np.linalg.eig(A)
>>> np.allclose(np.dot(A,V), np.dot(V,np.diag(e)))# V may not be unitary
True
>>> # Singular Value Decomposition
>>> U, s, V = np.linalg.svd(A, full_matrices = True)
>>> np.allclose(np.dot(np.dot(U, np.diag(s)),V), A)
True
10
新加坡国立大学商学院
Numerical Linear Algebra
• Solving Linear Systems
>>> np.linalg.solve(A, np.ones((5,1))).T
array([[ 0.16666667,
0.16666667,
0.16666667,
0.16666667,
0.16666667]])
-0.16666667,
-0.16666667,
-0.16666667,
0.83333333,
-0.16666667,
-0.16666667],
-0.16666667],
-0.16666667],
-0.16666667],
0.83333333]])
>>> np.linalg.inv(A)
array([[ 0.83333333,
[-0.16666667,
[-0.16666667,
[-0.16666667,
[-0.16666667,
-0.16666667,
0.83333333,
-0.16666667,
-0.16666667,
-0.16666667,
-0.16666667,
-0.16666667,
0.83333333,
-0.16666667,
-0.16666667,
# Solve Linear System
# Matrix Inverse
• Other Stuff
>>> np.linalg.norm(A)
6.324555320336759
>>> np.linalg.det(A)
5.9999999999999982
>>> np.trace(A)
10
>>> np.linalg.matrix_rank(A)
5
新加坡国立大学商学院
# Matrix norm
# Determinant (should be 6...)
# Matrix trace
# Matrix rank
11
Visualization (By Example)
• By now you should have a Python
distribution.
• Run IPython or Spyder
In [1]: %pylab
Using matplotlib backend: module://IPython.kernel.zmq.pylab.backend_inline
Populating the interactive namespace from numpy and matplotlib
In [2]: plot( arange(20) )
Out[2]: [<matplotlib.lines.Line2D at 0x8918850>]
12
新加坡国立大学商学院
Visualization (By Example)
• A Standard Brownian Motion
In [3]:
...:
...:
Out[3]:
dx = 0.01
# Hit CTRL-Enter for multi-line input
walk = (np.random.randn(1000) * np.sqrt(dx)).cumsum()
plot(np.arange(0,1000)*dx, walk - walk[0])
[<matplotlib.lines.Line2D at 0xd31f770>]
13
新加坡国立大学商学院
Visualization (By Example)
• A 2D Intensity Plot
In [4]:
...:
...:
...:
...:
...:
...:
Out[4]:
xy_extent = 20
xy_range = np.arange(-xy_extent,xy_extent,0.1)
X_m, Y_m = np.meshgrid(xy_range, xy_range)
f = ((X_m-5) - 2 * (Y_m+5)) ** 2 - (1.5*(X_m+3) + 1 * (Y_m-5)) ** 2
imshow(f, cmap = matplotlib.cm.bone, \
extent=[-xy_extent, xy_extent, -xy_extent, xy_extent])
colorbar()
<matplotlib.colorbar.Colorbar instance at 0x0C3AFE90>
14
新加坡国立大学商学院
Visualization (By Example)
• A 2D Contour Plot
In [5]:
...:
...:
...:
...:
...:
Out[5]:
xy_extent = 20
xy_range = np.arange(-xy_extent,xy_extent,0.1)
X_m, Y_m = np.meshgrid(xy_range, xy_range)
f = ((X_m-5) - 2 * (Y_m+5)) ** 2 - (1.5*(X_m+3) + 1 * (Y_m-5)) ** 2
contourf(X_m, Y_m, f, 20)
colorbar()
<matplotlib.colorbar.Colorbar instance at 0x10C12DA0>
15
新加坡国立大学商学院
Visualization (By Example)
Our Objective…
16
新加坡国立大学商学院
Visualization (By Example)
• Some data for plotting
In [6]:
...:
...:
...:
...:
XY = np.arange(1,1+100)/100.0
XZ = np.arange(1,1+1000)/1000.0
Y = np.random.randn(100)
Z = np.random.randn(1000)
Z.sort()
• Setting up the figure and subplots
In [7]:
...:
...:
...:
...:
...:
n_row = 2
n_col = 2
fig1 = plt.figure(figsize=(12,8))
axes = []; # Collect axis references in a list
for k in range(n_row * n_col):
axes.append( fig1.add_subplot(n_row ,n_col , k+1) )
17
新加坡国立大学商学院
Visualization (By Example)
• The simple line plot
In [8]:
...:
...:
...:
...:
...:
...:
Out[8]:
axes[0].plot( XZ - 0.5, np.cos(1.5 * (XZ - 0.5) * \
2 * 3.14159), color='k')
axes[0].set_xlim([-0.6, 0.6])
axes[0].set_title(r'$\cos\ 2\pi x$')
axes[0].set_xlabel('x')
axes[0].set_ylabel('y')
fig1
18
新加坡国立大学商学院
Visualization (By Example)
• The histogram of normal samples
In [9]: axes[1].hist( Z, bins=20, color='b')
...: axes[1].set_title('Some Histogram of 1000 samples from $N(0,1)$')
...: fig1
Out[9]:
19
新加坡国立大学商学院
Visualization (By Example)
• The scatter plots (with legend)
In [10]:
...:
...:
...:
...:
...:
Out[10]:
axes[2].scatter( XY[10:60], Y[:50] + XY[10:60]*0.05, marker='o', \
c=(0.5, 0.5, 0.5), label='Population 1')
axes[2].scatter( XY[40:90], Y[50:] + XY[40:90]*0.05, marker='x', \
c='#FF00A0', label='Population 2')
axes[2].legend(loc = 'best')
fig1
20
新加坡国立大学商学院
Visualization (By Example)
• The big mess of normal CDFs
In [11]:
...:
...:
...:
...:
...:
...:
...:
...:
...:
Out[11]:
axes[3].set_title(r'A Colourful Mess of $N(\mu_k, 1)$ CDFs')
col = ['r', 'g', 'b', 'c', 'm', 'y', 'k'] # w for white
line = ['solid', 'dashed', 'dashdot', 'dotted']
linestyle = [(c,l) for c in col for l in line]
for k in range(40):
ls_idx = k % len(linestyle)
axes[3].plot( Z+(k-20)*0.5, XZ, color = linestyle[ls_idx][0], \
linestyle=linestyle[ls_idx][1])
axes[3].text(-10, 0.45, 'Nothing to See Here.', fontsize=20)
fig1
21
新加坡国立大学商学院
Visualization (By Example)
22
新加坡国立大学商学院
Example: Ordinary Least Squares
• The standard linear model
– 𝑦𝑘 = 𝛽𝑇 𝑥𝑘 + 𝜀𝑘 where 𝜀𝑘 ~𝑁(0, 𝜎 2 )
– The Maximum Likelihood Estimator is the Least Squares
𝑇
2
Solution: 𝛽∗ = arg min 𝑁
𝑘=1 𝑦𝑘 − 𝛽 𝑥𝑘
𝛽
In [12]:
...:
...:
...:
...:
...:
...:
...:
...:
...:
...:
...:
import math
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
# Load data
(Choose between data sets with...
#
100, 200, 500, 1000, 5000, and 10000 samples)
data = np.load('OLS_data_500.npz')
X = data['X']
Y = data['Y']
true_beta = data['true_beta']
# Yeah... These are provided
true_error_std = data['true_error_std']
data.close()
23
新加坡国立大学商学院
Example: Ordinary Least Squares
In [13]: if X.shape[0] != Y.shape[0]:
# Always good to check for errors
...:
raise ValueError("Data dimension mismatch.")
...: num_pred = X.shape[1]
...: num_samples = X.shape[0]
In [14]:
...:
...:
...:
...:
...:
...:
...:
...:
...:
...:
...:
...:
# Descriptive Statistics
plt_side = math.ceil(math.sqrt(num_pred + 1))
fig = plt.figure(figsize = (12,12));
for k in range(num_pred + 1):
ax = fig.add_subplot(plt_side, plt_side, k+1)
if k == 0:
ax.hist(Y, bins=20)
ax.set_title( "Dependent Variable ($\\mu={mean:.3}$, $\\sigma={std:.3}$)".format( \
mean = np.mean(Y), std = np.std(Y)) )
else:
ax.hist(X[:, k-1], bins=20)
ax.set_title( "Predictor {pred} ($\\mu={mean:.3}$, $\\sigma={std:.3}$)".format( \
pred = k, mean = np.mean(X[:, k-1]), std = np.std(X[:, k-1])) )
24
新加坡国立大学商学院
Example: Ordinary Least Squares
25
新加坡国立大学商学院
Example: Ordinary Least Squares
In [15]: # OLS
...: XtX = np.dot(X.T, X)
...: XtY = np.dot(X.T, Y)
...: beta_est = np.linalg.solve(XtX, XtY)
...: s_sq = np.var(Y - np.dot(X, beta_est))
...:
...: print "True Betas", true_beta.T
...: print "Estimated Betas", beta_est.T
...: print
...:
...: print "True Error Variance: ", true_error_std ** 2
...: print "Estimated Error Variance: ", s_sq
True Betas [[ 1. 1. 1. 1. 1. 0.]]
Estimated Betas [[ 1.2389584
True Error Variance: 4
Estimated Error Variance:
0.98711812
0.90258976
0.86002269
0.99770273
0.01079312]]
3.85523406528
26
新加坡国立大学商学院
Example: Ordinary Least Squares
In [16]:
...:
...:
...:
...:
...:
...:
...:
...:
# Hypothesis Testing
standard_errors = np.reshape(np.sqrt(np.diag( \
s_sq * np.linalg.solve(XtX, np.eye(num_pred)) )), (num_pred,1))
print "Standard Errors: ", standard_errors.T
t_statistics = beta_est / standard_errors
print "t-Statistics: ", t_statistics.T
df = num_samples - num_pred
p_values = 2 * scipy.stats.t.cdf(-abs(t_statistics), df).T
print "p-values (2-sided): ", p_values
Standard Errors: [[ 0.26819354 0.05023788 0.16891535 0.09578419 0.09744182 0.06468454]]
t-Statistics: [[ 4.61964296 19.6488801
5.34344442
8.97875377 10.23895845 0.16685784]]
p-values (2-sided): [[ 4.91038992e-06 6.11554987e-64 1.39498275e-07 5.75446104e-18 1.92610049e-22 8.67550186e-01]]
So let’s do it again!
27
新加坡国立大学商学院
Example: Ordinary Least Squares
In [15]: # OLS Again
...: X = X[:,:-1]; num_pred = X.shape[1]; num_samples = X.shape[0]
...: XtX = np.dot(X.T, X)
...: XtY = np.dot(X.T, Y)
...: beta_est = np.linalg.solve(XtX, XtY)
...: s_sq = np.var(Y - np.dot(X, beta_est))
...: print "Estimated Betas", beta_est.T
...: standard_errors = np.reshape(np.sqrt(np.diag( \
...:
s_sq * np.linalg.solve(XtX, np.eye(num_pred)) )), (num_pred,1))
...: print "Standard Errors: ", standard_errors.T
...: t_statistics = beta_est / standard_errors
...: print "t-Statistics: ", t_statistics.T
...: df = num_samples - num_pred
...: p_values = 2 * scipy.stats.t.cdf(-abs(t_statistics), df).T
...: print "p-values (2-sided): ", p_values
Estimated Betas [[ 1.25027031 0.98744205 0.90124032 0.86084415 0.9975555 ]]
Standard Errors: [[ 0.25949091 0.05020176 0.16872633 0.09566025 0.09744054]]
t-Statistics: [[ 4.81816615 19.6694721
5.34143267
8.99897405 10.23758223]]
p-values (2-sided): [[ 1.92950811e-06 4.54116320e-64 1.40853414e-07 4.88551777e-18 1.93166121e-22]]
28
新加坡国立大学商学院
☺
29
新加坡国立大学商学院