SYSC4416: Deep Neural Networks
Winter 2026
Numerical Tutorial
Dr. Mohannad Abu Issa
Carleton University
Department of Systems and Computer Engineering
1. Network Definition
We consider a feedforward neural network with architecture
(2, 3, 3, 1)
which means:
• input layer: 2 features
• first hidden layer: 3 neurons
• second hidden layer: 3 neurons
• output layer: 1 neuron
We use the sigmoid activation function in all layers:
σ(z) =
1
1 + e−z
For simplicity, we use squared error:
1
J = (ŷ − y)2
2
2. Initial Parameters
We define the initial weights and biases as follows.
2.1 From input layer to hidden layer 1
0.5
0.2
W (1) = −0.3 0.8 ,
0.7 −0.1
2.2 From hidden layer 1 to hidden layer 2
0.4 −0.5 0.2
0.6 −0.2 ,
W (2) = 0.1
−0.3 0.4
0.5
1
0.1
b(1) = 0.2
−0.1
0.1
b(2) = −0.2
0.05
2.3 From hidden layer 2 to output layer
W (3) = 0.8 −0.4 0.3 ,
b(3) = 0.1
3. Core Equations
3.1 Forward propagation
z (2) = W (1) x + b(1) ,
a(2) = σ(z (2) )
z (3) = W (2) a(2) + b(2) ,
a(3) = σ(z (3) )
z (4) = W (3) a(3) + b(3) ,
3.2 Cost function
a(4) = ŷ = σ(z (4) )
1
J = (ŷ − y)2
2
3.3 Backpropagation For the output layer:
δ (4) = (a(4) − y)σ ′ (z (4) )
where
σ ′ (z) = a(1 − a)
For hidden layers:
δ (3) = (W (3) )T δ (4) ⊙ σ ′ (z (3) )
δ (2) = (W (2) )T δ (3) ⊙ σ ′ (z (2) )
3.4 Gradients
∂J
= δ (4) (a(3) )T ,
∂W (3)
∂J
= δ (3) (a(2) )T ,
∂W (2)
∂J
= δ (2) xT ,
∂W (1)
∂J
= δ (4)
∂b(3)
∂J
= δ (3)
∂b(2)
∂J
= δ (2)
∂b(1)
3.5 Gradient descent update With learning rate α = 0.1:
(l)
Wnew
= W (l) − α
(l)
b(l)
−α
new = b
2
∂J
∂W (l)
∂J
∂b(l)
4. Training Example 1
We take the first training sample:
x
(1)
1
=
,
0
y (1) = 1
4.1 Forward Propagation
Step 1: Hidden layer 1
z (2) = W (1) x(1) + b(1)
0.5
0.2 0.1
1
= −0.3 0.8
+ 0.2
0
0.7 −0.1
−0.1
0.5
0.1
0.6
= −0.3 + 0.2 = −0.1
0.7
−0.1
0.6
Apply sigmoid:
0.646
σ(0.6)
a(2) = σ(z (2) ) = σ(−0.1) ≈ 0.475
0.646
σ(0.6)
Step 2: Hidden layer 2
z (3) = W (2) a(2) + b(2)
0.1
0.646
0.4 −0.5 0.2
0.6 −0.2 0.475 + −0.2
= 0.1
0.05
0.646
−0.3 0.4
0.5
Each component is:
(3)
z1 = 0.4(0.646) − 0.5(0.475) + 0.2(0.646) + 0.1 = 0.2501
(3)
z2 = 0.1(0.646) + 0.6(0.475) − 0.2(0.646) − 0.2 = 0.0204
(3)
z3 = −0.3(0.646) + 0.4(0.475) + 0.5(0.646) + 0.05 = 0.3692
Thus,
0.250
z (3) ≈ 0.020
0.369
Apply sigmoid:
0.562
a(3) = σ(z (3) ) ≈ 0.505
0.591
3
Step 3: Output layer
z (4) = W (3) a(3) + b(3)
= 0.8(0.562) − 0.4(0.505) + 0.3(0.591) + 0.1
= 0.4496 − 0.202 + 0.1773 + 0.1 = 0.5249
Apply sigmoid:
ŷ (1) = a(4) = σ(0.5249) ≈ 0.628
4.2 Cost Function
1
J (1) = (ŷ (1) − y (1) )2
2
1
1
= (0.628 − 1)2 = (0.372)2 = 0.0692
2
2
4.3 Backpropagation
Step 1: Output layer error
δ (4) = (a(4) − y (1) )σ ′ (z (4) )
= (0.628 − 1) 0.628(1 − 0.628)
= (−0.372)(0.2336) ≈ −0.0869
Step 2: Hidden layer 2 error
δ (3) = (W (3) )T δ (4) ⊙ σ ′ (z (3) )
First,
0.8
−0.0695
(W (3) )T δ (4) = −0.4 (−0.0869) = 0.0348
0.3
−0.0261
Next,
0.562(1 − 0.562)
0.246
σ ′ (z (3) ) = 0.505(1 − 0.505) ≈ 0.250
0.591(1 − 0.591)
0.242
Thus,
−0.0695
0.246
−0.0171
δ (3) ≈ 0.0348 ⊙ 0.250 = 0.0087
−0.0261
0.242
−0.0063
Step 3: Hidden layer 1 error
δ (2) = (W (2) )T δ (3) ⊙ σ ′ (z (2) )
4
First,
0.4
0.1 −0.3
0.4
(W (2) )T = −0.5 0.6
0.2 −0.2 0.5
Then,
0.4
0.1 −0.3
−0.0171
−0.00408
0.4 0.0087 ≈ 0.01125
(W (2) )T δ (3) = −0.5 0.6
0.2 −0.2 0.5
−0.0063
−0.00831
Also,
0.646(1 − 0.646)
0.229
σ ′ (z (2) ) = 0.475(1 − 0.475) ≈ 0.249
0.646(1 − 0.646)
0.229
Therefore,
−0.00093
0.229
−0.00408
δ (2) ≈ 0.01125 ⊙ 0.249 = 0.00280
−0.00190
0.229
−0.00831
4.4 Gradient Descent Update
We use learning rate
α = 0.1
Gradients for output layer
∂J
= δ (4) (a(3) )T
(3)
∂W
= (−0.0869) 0.562 0.505 0.591 = −0.0488 −0.0439 −0.0514
∂J
= δ (4) = −0.0869
∂b(3)
Thus,
(3)
Wnew
= 0.8 −0.4 0.3 − 0.1 −0.0488 −0.0439 −0.0514
= 0.80488 −0.39561 0.30514
b(3)
new = 0.1 − 0.1(−0.0869) = 0.10869
Gradients for hidden layer 2
∂J
= δ (3) (a(2) )T
∂W (2)
5
−0.0171 = 0.0087 0.646 0.475 0.646
−0.0063
−0.0110 −0.0081 −0.0110
0.0041
0.0056
≈ 0.0056
−0.0041 −0.0030 −0.0041
−0.0171
∂J
= δ (3) ≈ 0.0087
∂b(2)
−0.0063
Hence,
0.4011 −0.4992 0.2011
(2)
0.5996 −0.2006
Wnew
≈ 0.0994
−0.2996 0.4003
0.5004
0.1017
b(2)
new ≈ −0.2009
0.0506
Gradients for hidden layer 1
∂J
= δ (2) (x(1) )T
∂W (1)
−0.00093
0
−0.00093 = 0.00280 1 0 = 0.00280 0
−0.00190 0
−0.00190
−0.00093
∂J
= δ (2) ≈ 0.00280
(1)
∂b
−0.00190
Thus,
0.500093
0.2
(1)
Wnew
≈ −0.300280 0.8
0.700190 −0.1
0.100093
0.199720
b(1)
new ≈
−0.099810
6
5. Training Example 2
Now we take a second training sample and repeat the full cycle using the updated parameters
from Example 1:
0
(2)
x =
,
y (2) = 0
1
We therefore use:
0.500093
0.2
0.100093
W (1) = −0.300280 0.8 ,
b(1) = 0.199720
0.700190 −0.1
−0.099810
0.4011 −0.4992 0.2011
0.1017
0.5996 −0.2006 ,
W (2) = 0.0994
b(2) = −0.2009
−0.2996 0.4003
0.5004
0.0506
W (3) = 0.80488 −0.39561 0.30514 ,
b(3) = 0.10869
5.1 Forward Propagation
Step 1: Hidden layer 1
z (2) = W (1) x(2) + b(1)
0.500093
0.2 0.100093
0
+ 0.199720
= −0.300280 0.8
1
0.700190 −0.1
−0.099810
0.300093
0.100093
0.2
= 0.8 + 0.199720 = 0.999720
−0.199810
−0.099810
−0.1
Apply sigmoid:
0.574
σ(0.300093)
a(2) ≈ σ(0.999720) ≈ 0.731
0.450
σ(−0.199810)
Step 2: Hidden layer 2
z (3) = W (2) a(2) + b(2)
Each component is:
(3)
z1 = 0.4011(0.574) − 0.4992(0.731) + 0.2011(0.450) + 0.1017 ≈ 0.0571
(3)
z2 = 0.0994(0.574) + 0.5996(0.731) − 0.2006(0.450) − 0.2009 ≈ 0.2044
(3)
z3 = −0.2996(0.574) + 0.4003(0.731) + 0.5004(0.450) + 0.0506 ≈ 0.3963
Thus,
0.0571
z (3) ≈ 0.2044
0.3963
7
Apply sigmoid:
0.514
a(3) ≈ 0.551
0.598
Step 3: Output layer
z (4) = W (3) a(3) + b(3)
= 0.80488(0.514) − 0.39561(0.551) + 0.30514(0.598) + 0.10869
≈ 0.4874
Hence,
ŷ (2) = σ(0.4874) ≈ 0.619
5.2 Cost Function
Since y (2) = 0,
1
1
J (2) = (ŷ (2) − 0)2 = (0.619)2 ≈ 0.1916
2
2
5.3 Backpropagation
Step 1: Output layer error
δ (4) = (a(4) − y (2) )σ ′ (z (4) )
= (0.619 − 0) 0.619(1 − 0.619)
= 0.619(0.2358) ≈ 0.1460
Step 2: Hidden layer 2 error
0.80488
0.1175
(W (3) )T δ (4) = −0.39561 (0.1460) ≈ −0.0578
0.30514
0.0446
Also,
0.514(1 − 0.514)
0.250
σ ′ (z (3) ) = 0.551(1 − 0.551) ≈ 0.247
0.598(1 − 0.598)
0.240
Thus,
0.1175
0.250
0.0294
δ (3) ≈ −0.0578 ⊙ 0.247 = −0.0143
0.0446
0.240
0.0107
8
Step 3: Hidden layer 1 error
0.4011
0.0994 −0.2996
0.4003
(W (2) )T = −0.4992 0.5996
0.2011 −0.2006 0.5004
0.0072
(W (2) )T δ (3) ≈ −0.0193
0.0144
Also,
0.574(1 − 0.574)
0.244
σ ′ (z (2) ) = 0.731(1 − 0.731) ≈ 0.197
0.450(1 − 0.450)
0.248
Hence,
0.00176
0.244
0.0072
δ (2) ≈ −0.0193 ⊙ 0.197 = −0.00380
0.00357
0.248
0.0144
5.4 Gradient Descent Update
Gradients for output layer
∂J
= δ (4) (a(3) )T
∂W (3)
= 0.1460 0.514 0.551 0.598 ≈ 0.0750 0.0804 0.0873
∂J
= 0.1460
∂b(3)
Update:
(3)
Wnew
≈ 0.80488 −0.39561 0.30514 − 0.1 0.0750 0.0804 0.0873
≈ 0.79738 −0.40365 0.29641
b(3)
new = 0.10869 − 0.1(0.1460) = 0.09409
Gradients for hidden layer 2
∂J
= δ (3) (a(2) )T
∂W (2)
0.0294 = −0.0143 0.574 0.731 0.450
0.0107
9
0.0169
0.0215
0.0132
≈ −0.0082 −0.0105 −0.0064
0.0061
0.0078
0.0048
0.0294
∂J
= −0.0143
∂b(2)
0.0107
Update:
0.3994 −0.5014 0.1998
(2)
0.6007 −0.2000
≈ 0.1002
Wnew
−0.3002 0.3995
0.4999
0.0988
b(2)
new ≈ −0.1995
0.0495
Gradients for hidden layer 1
∂J
= δ (2) (x(2) )T
∂W (1)
0.00176 0
0.00176
= −0.00380 0 1 = 0 −0.00380
0.00357
0 0.00357
0.00176
∂J
= −0.00380
∂b(1)
0.00357
Update:
0.500093
0.199824
(1)
Wnew
≈ −0.300280 0.800380
0.700190 −0.100357
0.099917
0.200100
b(1)
new ≈
−0.100167
10
6. Summary of the Two Training Cycles
• Sample 1:
x
(1)
1
=
,
0
y (1) = 1,
ŷ (1) ≈ 0.628,
J (1) ≈ 0.0692
x
(2)
0
=
,
1
y (2) = 0,
ŷ (2) ≈ 0.619,
J (2) ≈ 0.1916
• Sample 2:
• In each cycle, we performed:
– forward propagation
– cost computation
– backpropagation
– gradient descent parameter update
• This illustrates how a deep neural network learns by repeatedly computing predictions,
measuring error, propagating that error backward, and updating its weights and biases.
11
0
