Mathematics for Machine Learning
Multivariate Calculus
Formula sheet
Dr Samuel J. Cooper
Prof. David Dye
Dr A. Freddie Page
Derivatives of named functions
Definition of a derivative
df (x)
= lim
f (x) =
∆x→0
dx
0
f (x + ∆x) − f (x)
∆x
d 1
1
=− 2
dx x
x
d
(sin(x)) = cos(x)
dx
d
(cos(x)) = − sin(x)
dx
d
(exp(x)) = exp(x)
dx
Time saving rules
- Sum Rule:
d
d
d
(f (x) + g(x)) =
(f (x)) +
(g(x))
dx
dx
dx
- Power Rule:
Derivative structures
Given f (x) = axb ,
Given f = f (x, y, z)
then f 0 (x) = abx(b−1)
- Product Rule:
- Jacobian:
Given A(x) = f (x)g(x),
then A0 (x) = f 0 (x)g(x) + f (x)g 0 (x)
∂f ∂f ∂f
Jf =
,
,
∂x ∂y ∂z
- Chain Rule:
- Hessian:
Given h = h(p) and p = p(m),
dh
dh
dp
then
=
×
dm
dp dm


∂2f
∂x2


 2
∂ f
Hf = 
 ∂y∂x

 2
- Total derivative:
For the function f (x, y, z, ...), where each variable
is a function of parameter t, the total derivative is
∂ f
∂z∂x
df
∂f dx ∂f dy ∂f dz
=
+
+
+ ...
dt
∂x dt
∂y dt
∂z dt
1
∂2f
∂x∂y
∂2f
∂y 2
∂2f
∂z∂y
∂2f
∂x∂z



∂2f 
∂y∂z 


2
∂ f
∂z 2
Neural networks
- Grad :


- Activation function:
∂f
 ∂x 
 
 
∂f 
∇f = 
 ∂y 
 
 
ex − e−x
ex + e−x
4
d
1
=
(σ(x)) =
2
x
dx
cosh (x)
(e + e−x )2
σ(x) = tanh(x) =
∂f
∂z
- Directional Gradient:
Taylor Series
∇f.r̂
- Univariate:
- Gradient Descent:
1
f (x) = f (c) + f (c)(x − c) + f 00 (c)(x − c)2 + ...
2
∞
(n)
X
f (c)
=
(x − c)n
n!
n=0
0
sn+1 = sn − γ∇f
- Lagrange Multipliers λ:
∇f = λ∇g
- Multivariate:
- Least Squares - χ2 minimization:
f (x) = f (c) + Jf (c)(x − c)+
1
(x − c)t Hf (c)(x − c) + ...
2
χ2 =
n
X
(yi − y(xi ; ak ))2
i
criterion: ∇χ2 = 0
Optimization and Vector Calculus
- Newton-Raphson:
xi+1 = xi −
σi
anext = acur − γ∇χ2
n
X
(yi − y(xi ; ak )) ∂y
= acur + γ
σi
∂ak
i
f (xi )
f 0 (xi )
2