ML course 5 — Neural networks I:
feed-forward networks, backpropagation and training
Condensed course with foil/CFD applications
Program coverage. Week 2 (Thursday, ANN part) and week 3 (Tuesday): Herrmann et al. ch. 3.1–3.6; B&K videos 6.1–6.4.
Neurons, layers, weights, biases, activation functions and deep learning; forward propagation; backpropagation and automatic
differentiation; building and training networks in PyTorch; avoiding overfitting (hyperparameter tuning, cross-validation, early
stopping, regularization). Classification and advanced architectures are in course 6; solving ODEs with networks is in course 7.
Numbers and figures come from c5_examples.py.
1 Architecture
Neuron: a weighted sum of inputs plus a bias, passed through a nonlinear activation function σ: z = wT x + b,
a = σ(z).
Fully connected feed-forward network (multilayer perceptron, MLP) with L layers, input a(0) = x ∈ Rn0 :
z(l) = W(l) a(l−1) + b(l) ,
a(l) = σ(l) z(l) ,
l = 1, . . . , L,
ŷ = a(L) .
(1)
P
W(l) ∈ Rnl ×nl−1 are the weights and b(l) ∈ Rnl the biases; together they form θ, with l nl (nl−1 + 1) parameters. Layers 1..L − 1
are hidden; their number is the depth and nl the width. Deep learning = networks with many hidden layers, trained end-to-end
by gradient methods. The output activation depends on the task: identity for regression, softmax for classification (course 6).
z(1) = W(1) x + b(1) , a(1) = tanh(z(1) )
a(1)
1
x1
ŷ = z(2) = W(2) a(1) + b(2) (linear output)
ŷ
x2
a(1)
2
input
hidden (tanh)
C = (ŷ − y)2
output
Universal approximation (Cybenko 1989, Hornik 1991): one hidden layer with enough neurons and a non-polynomial
activation can approximate any continuous function on a compact set to any accuracy. The theorem says nothing about how
many neurons are needed, whether training finds them, or how the network behaves outside the data. Depth usually represents
functions far more efficiently than width.
1.1 Activation functions
Name
σ(z)
Sigmoid
Tanh
ReLU
Leaky ReLU
Softplus
SiLU / GELU
1/(1 + e )
tanh z
max(0, z)
max(βz, z)
ln(1 + ez )
z sigmoid(z), z Φ(z)
−z
σ′ (z)
Properties
σ(1 − σ)
1 − tanh2 z
1 if z > 0, else 0
1 or β
sigmoid
smooth
output in (0, 1); saturates (σ′ ≤ 0.25) ⇒ vanishing gradients
zero-centred, smooth (C ∞ ); common in PINNs
cheap, no saturation for z > 0; “dead” neurons; σ′′ = 0 a.e.
keeps a small gradient for z < 0
smooth ReLU
smooth ReLU-like; modern default
(z)
3
0
1.0
sigmoid
tanh
ReLU
SiLU
2
1
0.5
0
1
(z)
0.0
4
2
0
z
2
4
4
2
0
z
2
4
Without nonlinear activations, any stack of layers collapses to a single linear map. For models whose derivatives enter the loss
(PINNs, course 7), use smooth activations (tanh, SiLU): with ReLU, second derivatives vanish.
2 Forward propagation and backpropagation
Forward pass: evaluate (1) layer by layer and store every z(l) and a(l) . For a batch, stack samples as columns: Z(l) =
W(l) A(l−1) + b(l) 1T , so a whole layer is one matrix product.
ML course 5 — Neural networks I: feed-forward networks, backpropagation, training
1
Backpropagation is the chain rule organized backwards (reverse-mode AD, course 3). Define the error signal δ(l) = ∂C/∂z(l) .
Then
∂C
⊙ σ(L)′ z(L)
∂a(L)
δ(l) = W(l+1)T δ(l+1) ⊙ σ(l)′ z(l) , l = L − 1, . . . , 1
∂C
∂C
= δ(l) a(l−1)T ,
= δ(l)
∂W(l)
∂b(l)
δ(L) =
(output layer)
(2)
(propagate backwards)
(3)
(gradients)
(4)
One forward and one backward pass give all gradients, at roughly twice the cost of a forward pass. For a batch, sum
(average) the per-sample gradients.
Vanishing/exploding gradients. δ(1) contains the product of L factors W(l)T diag(σ′ ). If they are consistently < 1 (saturated
sigmoids), gradients vanish in early layers; if > 1, they explode. Remedies: suitable activations; initialization that keeps
variances constant (Glorot/Xavier: Var W = 2/(nin + nout ) for tanh; He: 2/nin for ReLU); normalization layers; residual
connections.
Foil/CFD corner: forward and backward pass by hand
Network 2–2–1 (figure above), x = (1, 0.5), target y = 0.5, W(1) =
"
0.5
0.8
#
−0.3 (1)
, b = (0.1, −0.1), W(2) = (1.0, −1.5),
0.2
b(2) = 0.2.
• Forward: z(1) = (0.5 − 0.15 + 0.1, 0.8 + 0.1 − 0.1) = (0.45, 0.80); a(1) = tanh z(1) = (0.4219, 0.6640); ŷ =
0.4219 − 1.5 · 0.6640 + 0.2 = −0.3742; C = (ŷ − y)2 = 0.7641.
• Output: δ(2) = 2(ŷ − y) = −1.7483 ⇒ ∂C/∂W(2) = δ(2) a(1) = (−0.7376, −1.1609), ∂C/∂b(2) = −1.7483.
• Hidden: δ(1) = W(2)T δ(2) ⊙ (1 − a(1)2
2.6225 · 0.5591) = (−1.4371, 1.4661).
" ) = (−1.7483 · 0.8220,
#
−1.4371 −0.7186
(1)
(1) T
• Gradients: ∂C/∂W = δ x =
, ∂C/∂b(1) = δ(1) .
1.4661
0.7331
The script repeats this in PyTorch: loss.backward() returns exactly the same numbers.
3 Training networks in PyTorch
import torch
torch.set_default_dtype(torch.float64)
net = torch.nn.Sequential(torch.nn.Linear(3, 16), torch.nn.Tanh(),
# inputs: alpha, ln(h/c), AR
torch.nn.Linear(16, 16), torch.nn.Tanh(),
torch.nn.Linear(16, 1))
# output: C_L (scaled)
opt = torch.optim.Adam(net.parameters(), lr=3e-3, weight_decay=0.0)
best, wait = float("inf"), 0
for epoch in range(4000):
net.train(); opt.zero_grad()
loss = torch.mean((net(X_tr) - y_tr)**2); loss.backward(); opt.step()
net.eval()
with torch.no_grad(): val = torch.mean((net(X_va) - y_va)**2).item()
if val < best: best, wait, state = val, 0, {k: v.clone() for k, v in net.state_dict().items()}
else:
wait += 1
if wait > 300: break
# early stopping with patience 300
net.load_state_dict(state)
# restore the best-validation weights
Checklist. Scale inputs and outputs to O(1), using training-set statistics only. Use float64 for smooth regression and scientific
problems. Shuffle the data. Watch training and validation curves. Initialize with a fixed seed for reproducibility. Report the test
error of the final model only once.
3.1 Hyperparameters and how to tune them
Architecture (depth, width, activation), optimizer (type, learning rate, schedule, batch size), regularization (weight decay λ,
dropout rate, patience) and the number of epochs are all hyperparameters. Tune them on validation data or by k-fold CV
(course 4), with grid search (few parameters), random search (more efficient when only some parameters matter) or Bayesian
optimization. Among models that are equally good within the noise, prefer the simplest.
3.2 Avoiding overfitting
• More / better data: the most effective remedy; for CFD data, a space-filling design (Latin hypercube) in the input space.
• Smaller model or regularization: weight decay (the L2 penalty λ∥θ∥2 ), dropout (course 6), physics terms (course 7).
• Early stopping: monitor the validation loss and keep the weights at its minimum. Training is then implicitly regularized,
because weights start small and grow with training time.
• Cross-validation for the final choice of hyperparameters when data are scarce.
ML course 5 — Neural networks I: feed-forward networks, backpropagation, training
2
Foil/CFD corner: an MLP surrogate for lift near the free surface
Data: 260 cases computed with the vortex-lattice method with free-surface images (high-Froude limit) from the LLT/VLM
course, random over α ∈ [−4◦ , 10◦ ], h/c ∈ [0.3, 3] (log-uniform), AR ∈ [3, 8], with added noise σ = 0.005 in C L . Inputs
(α, ln(h/c), AR) standardized; split 160/50/50. Tanh MLPs, Adam, early stopping (patience 300).
(width, depth)
(4,1)
(16,1)
(64,1)
(4,2)
(16,2)
(64,2)
(4,3)
(16,3)
(64,3)
parameters
train RMSE
val RMSE
21
.0075
.0083
81
.0039
.0059
321
.0036
.0057
41
.0054
.0069
353
.0038
.0052
4481
.0038
.0052
61
.0070
.0080
625
.0038
.0052
8641
.0040
.0051
Width 4 underfits. From (16,2) on, the validation RMSE sits at the noise level (≈ 0.005), so the simplest of these, (16,2), is
the sensible choice. The formally best (64,3) has test RMSE 0.0055 in C L .
MSE (scaled)
10 1
CL at = 5 , AR = 6
early-stopping
point
10 3
10 5
training (25 samples)
validation
validation, weight decay
10 7
0
VLM + images (reference)
MLP 64x3
0.36
0.34
0.32
0.30
0.28
1000 2000 3000 4000 5000 6000
epoch
0.3
0.5
1
submergence h/c
2
3
Left: overfitting. A 64×3 network (8641 parameters) trained on only 25 samples drives the training MSE to 10−7 . The
validation MSE is lowest at epoch 864 (1.7 × 10−3 ) and then rises to 4.5 × 10−3 : early stopping keeps the epoch-864 weights.
Weight decay (10−2 ) instead holds the training MSE near 2 × 10−3 and the validation MSE at 3.5 × 10−3 . The spikes are
Adam instabilities at a very small loss (a learning-rate decay would remove them). Right: the trained surrogate reproduces
the VLM lift loss towards the surface at α = 5◦ , AR = 6 within the noise level.
3.3 Design choices, measured
Same data, 16×2 network, same seed; only one choice is changed at a time (RMSE in C L ; noise level 0.005):
Variant
tanh, standardized inputs, Adam + early stopping
SiLU
sigmoid
ReLU
unscaled inputs (α in degrees, AR raw)
plain SGD, η = 0.05, 4000 full-batch steps
L-BFGS to convergence, no early stopping
train RMSE
val RMSE
0.0038
0.0037
0.0038
0.0040
0.0041
0.0057
0.0003
0.0052
0.0054
0.0059
0.0078
0.0060
0.0079
0.0170
Comment
reference
smooth: equivalent
slightly worse (not zero-centred)
piecewise-linear fit of a smooth function
mild here (inputs O(1–10)); severe for Re ∼ 106
not yet converged
overfits: training error 17× below the noise level
Lessons. For smooth physical responses, prefer smooth activations. Scale inputs, especially quantities spanning decades (Re, Fr,
h/c: use logarithms). A powerful optimizer without a stopping criterion simply fits the noise, so early stopping or regularization
must accompany it.
Gradient check. When writing backpropagation yourself, compare with central differences, ∂C/∂θ j ≈ [C(θ + he j ) − C(θ −
he j )]/2h with h ≈ 10−5 , on a few random parameters (course 3). Relative agreement of 10−7 or better (float64) means the
implementation is right.
ML course 5 — Neural networks I: feed-forward networks, backpropagation, training
3
4 Summary and learning-outcome checklist
Learning outcome (program)
Key result
Sect.
Describe ANNs: layers, neurons, activations, deep learning
Define biases, weights, activation functions
Describe and compute the forward pass
Back-propagation to find gradients
Apply AutoGrad for these gradients
Build and train ANNs in PyTorch; regression
Avoid overfitting: tuning, CV, early stopping, regularization
a(l) = σ(W(l) a(l−1) + b(l) ); activation table
1
W(l) , b(l) , σ; parameter count
1
Layer-by-layer evaluation; hand example
δ(L) , δ(l) = (WT δ) ⊙ σ′ , ∇W = δaT
loss.backward() matches the hand computation
nn.Sequential, optim, training loop
2
2
2, 3
3
Validation-based choices, patience, weight decay
3.1–
3.2
Exam-style self-check. (1) Count the parameters of a 3–16–16–1 network. (2) Derive δ(l) from δ(l+1) with the chain rule. (3)
Why does a network without activation functions reduce to linear regression? (4) Why are sigmoid networks prone to vanishing
gradients? (5) What exactly does early stopping return, and why is it a regularizer? (6) A surrogate trained on h/c ∈ [0.3, 3] is
queried at h/c = 0.1: what do you expect?
References. Herrmann et al., Deep Learning in Computational Mechanics, 2nd ed., 2025, ch. 3. Brunton & Kutz, Data-Driven
Science and Engineering, ch. 6. I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, MIT Press, 2016, ch. 6–8. X.
Glorot & Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks”, AISTATS, 2010. K. Hornik,
“Approximation capabilities of multilayer feedforward networks”, Neural Networks 4, 1991.
ML course 5 — Neural networks I: feed-forward networks, backpropagation, training
4