Lecture 3: Math Primer II
Machine Learning
Andrew Rosenberg
Today
•
•
•
•
Wrap up of probability
Vectors, Matrices.
Calculus
Derivation with respect to a vector.
1
Properties of probability density
functions
Sum Rule
Product Rule
2
Expected Values
• Given a random variable, with a distribution
p(X), what is the expected value of X?
3
Multinomial Distribution
• If a variable, x, can take 1-of-K states, we
represent the distribution of this variable
as a multinomial distribution.
• The probability of x being in state k is μk
4
Expected Value of a Multinomial
• The expected value is the mean values.
5
Gaussian Distribution
• One Dimension
• D-Dimensions
6
Gaussians
7
How machine learning uses
statistical modeling
• Expectation
– The expected value of a function is the
hypothesis
• Variance
– The variance is the confidence in that
hypothesis
8
Variance
• The variance of a random variable describes
how much variability around the expected
value there is.
• Calculated as the expected squared error.
9
Covariance
• The covariance of two random variables
expresses how they vary together.
• If two variables are independent, their
covariance equals zero.
10
Linear Algebra
• Vectors
– A one dimensional array.
– If not specified, assume x is a column
vector.
• Matrices
– Higher dimensional array.
– Typically denoted with capital letters.
– n rows by m columns
11
Transposition
• Transposing a matrix swaps columns and
rows.
12
Transposition
• Transposing a matrix swaps columns and
rows.
13
Addition
• Matrices can be added to themselves iff
they have the same dimensions.
– A and B are both n-by-m matrices.
14
Multiplication
• To multiply two matrices, the inner dimensions must
be the same.
– An n-by-m matrix can be multiplied by an m-by-k matrix
15
Inversion
• The inverse of an n-by-n or square matrix
A is denoted A-1, and has the following
property.
• Where I is the identity matrix is an n-by-n
matrix with ones along the diagonal.
– Iij = 1 iff i = j, 0 otherwise
16
Identity Matrix
• Matrices are invariant under multiplication
by the identity matrix.
17
Helpful matrix inversion properties
18
Norm
• The norm of a vector, x, represents the
euclidean length of a vector.
19
Positive Definite-ness
• Quadratic form
– Scalar
– Vector
• Positive Definite matrix M
• Positive Semi-definite
20
Calculus
• Derivatives and Integrals
• Optimization
21
Derivatives
• A derivative of a function defines the
slope at a point x.
22
Derivative Example
23
Integrals
• Integration is the inverse operation of
derivation (plus a constant)
• Graphically, an integral can be considered
the area under the curve defined by f(x)
24
Integration Example
25
Vector Calculus
• Derivation with respect to a matrix or
vector
• Gradient
• Change of Variables with a Vector
26
Derivative w.r.t. a vector
• Given a vector x, and a function f(x), how
can we find f’(x)?
27
Derivative w.r.t. a vector
• Given a vector x, and a function f(x), how
can we find f’(x)?
28
Example Derivation
29
Example Derivation
Also referred to as the gradient of a function.
30
Useful Vector Calculus identities
• Scalar Multiplication
• Product Rule
31
Useful Vector Calculus identities
• Derivative of an inverse
• Change of Variable
32
Optimization
• Have an objective function that we’d like to
maximize or minimize, f(x)
• Set the first derivative to zero.
33
Optimization with constraints
• What if I want to constrain the parameters
of the model.
– The mean is less than 10
• Find the best likelihood, subject to a
constraint.
• Two functions:
– An objective function to maximize
– An inequality that must be satisfied
34
Lagrange Multipliers
• Find maxima of
f(x,y) subject to a
constraint.
35
General form
• Maximizing:
• Subject to:
• Introduce a new variable, and find a
maxima.
36
Example
• Maximizing:
• Subject to:
• Introduce a new variable, and find a
maxima.
37
Example
Now have 3 equations with 3 unknowns.
38
Example
Eliminate Lambda
Substitute and Solve
39
Why does Machine Learning need
these tools?
• Calculus
– We need to identify the maximum likelihood, or
minimum risk. Optimization
– Integration allows the marginalization of
continuous probability density functions
• Linear Algebra
– Many features leads to high dimensional spaces
– Vectors and matrices allow us to compactly
describe and manipulate high dimension al
feature spaces.
40
Why does Machine Learning need
these tools?
• Vector Calculus
– All of the optimization needs to be performed
in high dimensional spaces
– Optimization of multiple variables
simultaneously – Gradient Descent
– Want to take a marginal over high
dimensional distributions like Gaussians.
41
Next Time
• Linear Regression
– Then Regularization
42