Machine Learning and AI with Simple Python and Matlab Scripts
IEEE Press
445 Hoes Lane
Piscataway, NJ 08854
IEEE Press Editorial Board
Sarah Spurgeon, Editor-in-Chief
Moeness Amin
Jón Atli Benediktsson
Adam Drobot
James Duncan
Ekram Hossain
Brian Johnson
Hai Li
James Lyke
Joydeep Mitra
Desineni Subbaram Naidu
Tony Q. S. Quek
Behzad Razavi
Thomas Robertazzi
Patrick Chik Yue
Machine Learning and AI with Simple Python and
Matlab Scripts
Courseware for Non-computing Majors
M. Ümit Uyar
The City College of New York
New York, USA
Copyright © 2025 by The Institute of Electrical and Electronics Engineers, Inc. All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section
107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or
authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222
Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com.
Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons,
Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/
go/permission.
The manufacturer’s authorized representative according to the EU General Product Safety Regulation is Wiley-VCH
GmbH, Boschstr. 12, 69469 Weinheim, Germany, e-mail: Product_Safety@wiley.com.
Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or
its affiliates in the United States and other countries and may not be used without written permission. All other
trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product
or vendor mentioned in this book.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing
this book, they make no representations or warranties with respect to the accuracy or completeness of the contents
of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose.
No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies
contained herein may not be suitable for your situation. You should consult with a professional where appropriate.
Further, readers should be aware that websites listed in this work may have changed or disappeared between when
this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or
any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer
Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317)
572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be
available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data Applied for
Hardback ISBN: 9781394294954
Cover Design: Wiley
Cover image is a painting titled “Yeni Arkadaş” (New Friend) by Mümtaz Yener (private collection of his daugther
Prof. Dr. Göksun Nina Say-Yener)
Author Photo: Courtesy of M. Ümit Uyar
Set in 9.5/12.5pt STIXTwoText by Straive, Chennai, India
To my beloved daughters
Aylin Emine and Melisa Ayşe
vii
Contents
About the Author xiii
Preface xv
Acknowledgments xvii
About the Companion Website
xix
1
1.1
1.2
1.3
1.4
1.5
Introduction 1
Artificial Intelligence 1
A Historical Perspective 1
Principles of AI 2
Applications That Are Impossible Without AI
Organization of This Book 3
2
2.1
2.2
2.3
2.3.1
2.3.2
2.3.3
2.3.4
2.4
2.5
2.5.1
2.6
2.6.1
2.6.2
2.6.3
2.7
2.8
2.9
2.10
2.11
2.12
2.13
Artificial Neural Networks 7
Introduction 7
Applications of ANNs 7
Components of ANNs 8
Neurons 8
Sigmoid Activation Function 9
Rectilinear Activation Function 9
Weights of Synapses 10
Training an ANN 11
Forward Propagation 12
Forward Propagation from Input to Hidden Layer 13
Back Propagation 13
Back Propagation for a Neuron 13
Back Propagation – from Output to Hidden Layer 15
Back Propagation – from Hidden Layer to Input 16
Updating Weights 17
ANN with Input Bias 17
A Simple Algorithm for ANN Training 18
Computational Complexity of ANN Training 18
Normalization of ANN Inputs and Outputs 19
Concluding Remarks 20
Exercises for Chapter 2 20
2
viii
Contents
3
3.1
3.2
3.3
3.4
3.5
3.5.1
3.5.2
3.5.3
3.5.4
3.5.5
3.5.6
3.5.7
3.5.8
3.5.9
3.5.10
3.6
3.6.1
3.6.2
3.7
3.8
ANNs for Optimized Prediction 23
Introduction 23
Selection of ANN Inputs 24
Selection of ANN Outputs 24
Construction of Hidden Layers 25
Case Study 1: Sleep-Study Example 25
Using Matrices for ANN Training 26
Forward Propagation 28
Back Propagation 28
Updating Weights 29
Forward Propagation with New Weights 29
Back Propagation with New Weights 30
Using Normalized Input and Output Values 31
Reducing Errors During Training 34
Implementation of Sleep-Study ANN in Python 34
Implementation of Sleep-Study ANN in Matlab 37
Case Study 2: Prediction of Bike Rentals 41
Python Script for Bike Rentals Using an ANN 41
Matlab Script for Bike Rentals Using an ANN 46
Concluding Remarks 48
Exercises for Chapter 3 48
4
4.1
4.2
4.3
4.3.1
4.3.2
4.3.3
4.3.4
4.4
4.5
4.5.1
4.5.2
4.6
4.7
4.8
4.9
ANNs for Financial Stock Trading 51
Introduction 51
Programs that Buy and Sell Stocks 51
Technical Indicators 51
Simple Moving Average 52
Momentum 53
Exponential Moving Average 54
Bollinger Bands 54
A Simple Algorithmic Trading Policy 55
A Simple ANN for Algorithmic Stock Trading 57
ANN Inputs and Outputs 57
ANN Architecture 58
Python Script for Stock Trading Using an ANN 59
Matlab Script for Stock Trading Using an ANN 63
Concluding Remarks 65
Exercises for Chapter 4 65
5
5.1
5.2
5.3
5.4
5.5
5.6
5.7
ANNs for Alzheimer’s Disease Prognosis 67
Introduction 67
Alzheimer’s Disease 67
A Simple ANN for AD Prognosis 68
Python Script for AD Prognosis Using an ANN 71
Matlab Script for AD Prognosis Using an ANN 75
Concluding Remarks 80
Exercises for Chapter 5 81
Contents
6
6.1
6.2
6.3
6.3.1
6.3.2
6.3.3
6.3.4
6.4
6.5
6.6
6.7
ANNs for Natural Language Processing 83
Introduction 83
Impact of Text Messages on Stock Markets 84
A Simple ANN for NLP 85
ANN Inputs and Outputs 85
Keywords 85
Formation of Training Data 86
ANN Architecture 88
Python Script for NLP Using an ANN 89
Matlab Script for NLP Using an ANN 92
Concluding Remarks 96
Exercises for Chapter 6 97
7
7.1
7.1.1
7.2
7.3
7.4
7.5
7.6
7.7
7.8
Convolutional Neural Networks 99
Introduction 99
Training CNNs 100
Variations of CNNs 101
Applications of CNNs 101
CNN Components 102
A Numerical Example of a CNN 102
Computational Cost of CNN Training 108
Concluding Remarks 112
Exercises for Chapter 7 112
8
8.1
8.2
8.3
8.4
8.5
8.6
8.7
CNNs for Optical Character Recognition 115
Introduction 115
A Simple CNN for OCR 115
Organization of Training and Reference Files 117
Python Script for OCR Using a CNN 119
Matlab Script for OCR Using a CNN 124
Concluding Remarks 130
Exercises for Chapter 8 130
9
9.1
9.2
9.3
9.4
9.5
9.6
9.7
CNNs for Speech Recognition 133
Introduction 133
A Simple CNN for Speech Recognition 134
Organization of Training and Reference Files 136
Python Script for Speech Recognition Using a CNN 138
Matlab Script for Speech Recognition Using a CNN 144
Concluding Remarks 150
Exercises for Chapter 9 150
10
10.1
10.2
10.2.1
10.2.2
Recurrent Neural Networks 151
Introduction 151
One-to-One Single RNN Cell 153
A Simple Alphabet and One-Hot Encoding 156
Forward and Back Propagation 157
ix
x
Contents
10.3
10.4
10.5
10.5.1
10.5.2
10.6
10.7
A Numerical Example 158
Multiple Hidden Layers 163
Embedding Layer 165
Forward and Back Propagation with Embedding 167
A Numerical Example with Embedding 168
Concluding Remarks 172
Exercises for Chapter 10 172
11
11.1
11.2
11.3
11.4
11.5
11.6
11.7
RNNs for Chatbot Implementation 175
Introduction 175
Many-to-Many RNN Architecture 175
A Simple Chatbot 176
Python Script for a Chatbot Using an RNN 179
Matlab Script for a Chatbot Using an RNN 183
Concluding Remarks 188
Exercises for Chapter 11 189
12
12.1
12.2
12.3
12.4
12.5
12.6
12.7
12.8
RNNs with Attention 191
Introduction 191
One-to-One RNN Cell with Attention 191
Forward and Back Propagation 193
A Numerical Example 195
Embedding Layer 200
A Numerical Example with Embedding 202
Concluding Remarks 207
Exercises for Chapter 12 207
13
13.1
13.2
13.3
13.4
13.5
13.6
RNNs with Attention for Machine Translation 209
Introduction 209
Many-to-Many Architecture 210
Python Script for Machine Translation by an RNN-Att 211
Matlab Script for Machine Translation by an RNN-Att 216
Concluding Remarks 223
Exercises for Chapter 13 223
14
14.1
14.2
14.3
14.4
14.5
14.6
14.7
Genetic Algorithms 225
Introduction 225
Genetic Algorithm Elements 226
A Simple Algorithm for a GA 227
An Example of a GA 230
Convergence in GAs 231
Concluding Remarks 232
Exercises for Chapter 14 232
15
15.1
15.2
GAs for Dietary Menu Selection 235
Introduction 235
Definition of the KP 236
Contents
15.3
15.4
15.5
15.6
15.7
15.8
15.9
A Simple Algorithm for the KP 238
Variations of the KP 239
GAs for KP Solution 240
Python Script for Dietary Menu Selection Using a GA 242
Matlab Script for Dietary Menu Selection Using a GA 245
Concluding Remarks 248
Exercises for Chapter 15 248
16
16.1
16.2
16.3
16.4
16.4.1
16.4.2
16.4.3
16.5
16.6
16.7
16.8
GAs for Drone Flight Control 251
Introduction 251
UAV Swarms 251
UAV Flight Control 252
A Simple GA for UAV Flight Control 253
Virtual Force-Based Fitness Function 254
FGA Progression 255
Chromosome for FGA 257
Python Script for UAV Flight Control Using a GA 260
Matlab Script for UAV Flight Control Using a GA 264
Concluding Remarks 270
Exercises for Chapter 16 271
17
17.1
17.2
17.3
17.4
17.5
17.6
17.7
17.8
17.9
GAs for Route Optimization 273
Introduction 273
Definition of the TSP 274
A Simple Algorithm for the TSP 276
Variations of the TSP 277
GA Solution for the TSP 277
Python Script for Route Optimization Using a GA 279
Matlab Script for Route Optimization Using a GA 284
Concluding Remarks 287
Exercises for Chapter 17 289
18
18.1
18.2
18.2.1
18.2.2
18.2.3
18.3
18.3.1
18.3.2
18.3.3
18.3.4
18.3.5
18.4
18.4.1
18.4.2
Evolutionary Methods 291
Introduction 291
Particle Swarm Optimization 291
Applications of PSO 292
PSO Operation 293
Remarks for PSO 298
Differential Evolution 298
Different Versions of DE 299
Applications of DE 299
A Simple Algorithm for DE 299
Numerical Example: Maximum of sinc by DE 302
Remarks for DE 305
Grammatical Evolution 306
A Simple Algorithm for GE 306
Definition of GE 307
xi
xii
Contents
18.4.3
18.4.4
A Simple GA to Implement GE 314
Remarks on GE 315
Appendix A ANNs with Bias 317
A.1
Introduction 317
A.2
Training with Bias Input 317
A.3
Forward Propagation 318
A.3.1
Forward Propagation from Input to Hidden Layer 319
A.3.2
Neuron Back Propagation with Bias Input 319
Appendix B Sleep Study ANN with Bias 321
B.1
Inclusion of Bias Term in ANN 321
B.1.1
Inclusion of Bias in Matrices 321
B.1.2
Forward Propagation with Biases 322
Appendix C Back Propagation in a CNN
327
Appendix D Back Propagation Through Time in an RNN
D.1
Back Propagation in an RNN 331
D.2
Embedding Layer 335
331
Appendix E Back Propagation Through Time in an RNN with Attention 337
E.1
Back Propagation in an RNN-Att 337
E.2
Embedding Layer 340
Bibliography 343
Index 353
xiii
About the Author
Dr. M. Ümit Uyar is a professor at the City College of the City
University of New York. He is the co-founder and current director
of the Computer Engineering program at the City College. He was
the lead and co-principal investigator for large grants to conduct
research on AI and game theory-based autonomous software agents,
knowledge sharing mobile agents using bio-inspired algorithms for
topology control in mobile networks, tactical interoperability of
combat networks and efficient reliable end-to-end communications.
Based on his research experience in civilian and military telecommunication protocols and mobile ad hoc computer networks, Dr. Uyar
developed AI and game theory algorithms for large swarms of autonomous drones and intelligent
techniques for personalized prognosis of degenerative diseases such as Alzheimer’s disease and
cancer. While in industry, he developed widely used software tools that improved the software
development cycle by orders of magnitude. He has published extensively in AI, game theory and
formal description techniques used for complex telecommunication protocols, edited books and
co-chaired international conferences.
Prior to joining academia, he was a Distinguished Member of Technical Staff at AT&T Bell Labs.
He is an IEEE Fellow and holds seven US patents. Dr. Uyar has a BS degree from İstanbul Teknik
Üniversitesi, and MS and PhD degrees from Cornell University, all in electrical engineering.
xv
Preface
This book is an introduction to artificial intelligence (AI)-based problem solving techniques,
together with the basic concepts they originated from and their computational principles.
Readers will see how several real-life problems can be modelled so that AI methods can be applied
to solve them.
Each chapter focuses on a realistic challenge but with reduced dimensions, presenting a simple, easy-to-follow solution as both Python and Matlab scripts. Projects include making financial
stock market predictions based on text messages from famous people, algorithmic trading of financial assets, bike rental predictions for the City of London, personalized prediction of Alzheimer’s
disease based on genetic information, speech recognition, chatbot implementation and translation
of written text from one language to another.
The emphasis on these projects is not to write programs to implement the AI algorithms in
computer languages but rather to demonstrate the relative simplicity of the AI algorithms used
to solve these problems.
After finishing this book, it will become apparent to the reader that although much has been
accomplished in the world of AI, there is still much more to be learned. This book will prepare
them for tackling a wide range of complex problems by applying cutting-edge AI techniques and
advancing on new frontiers.
About the cover: Yeni Arkadaş (New Friend) by Mümtaz Yener (private collection of his
daughter Prof. Dr. Nina Göksun Say-Yener).
Mümtaz Yener (1918–2007) worked and lived in Istanbul as a painter with his best friend
and wife Şadan Yener. His extensive artwork has been critically acclaimed around the world
and featured in renowned museum exhibitions and private art collections in Turkey, Europe, the
United States, Japan and Brazil. He believed that “freedom depends on the amount of
domination that humans have over machines.”
xvii
Acknowledgments
I am grateful for the comments that I received about the materials presented in this book from
the undergraduate and graduate students who have taken my classes at the City University of New
York over the years. Their relentless pursuit of knowledge has always been refreshing. Specifically, I
acknowledge the efforts of the following past and present students of the City College of New York:
Olga Chsherbakova for speedy implementation of Python and Matlab scripts
Hasan Şuca Kayman for his skill in deriving complex gradients and implementing scripts for
recurrent neural network and attention models
● Dr. Samrat S. Batth for contributions to the chapter on financial applications
● Michal Kropiewnicki for critical reading and constructive comments
● Kelvin Ma for his contributions to the manuscript and projects
● Clement McClean for preparing student assignments with admirable ease
● Grace McGrath, Joe Malubay, Jian Wen Choong, Ishmam Fardin and Sultana Begum for contributions to the Alzheimer’s disease prognosis scripts
● Ricardo Valdez for his contributions to the manuscript and projects and for his derivation of ANN
examples with an impressive show of patience
● Alexander Verzun and Roberto Behar for corrections to the text and scripts
●
●
I thank my friend and colleague Prof. Janusz Kusyk for his invaluable contributions on all
concepts in this book throughout the chapters.
I thank Prof. Dr. Nina Göksun Say-Yener for generously sharing Mümtaz Yener’s masterpiece
Yeni Arkadaş (New Friend) as the cover of this book.
I thank the Wiley team for their support and understanding: Mary Hatcher, Brett Kurzman, Ryan
Coach, Dimple Philip, Vijayalakshmi Saminathan and Akhil Ajikumar. Special thanks go to Chris
Cartwright for his meticulous editing.
Lastly, I would like to thank Aylin E. Uyar for simplifying complex concepts covered throughout and making them more accessible to a wider audience, including readers without a software
engineering background.
August 2024
M. Ümit Uyar, PhD
The City College of New York, New York, USA
xix
About the Companion Website
This book is accompanied by a companion website:
www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
This website includes an instructor companion site with Python and Matlab code and scripts.
1
1
Introduction
1.1 Artificial Intelligence
Artificial intelligence (AI) has become one of the most exciting and active fields of research in recent
years. AI attempts to develop software and computing devices that are able to perform tasks which
have been associated only with thinking beings for centuries. Mimicking intelligence, creativity,
deduction capability and ability to learn from experience are some of the directions that AI undertakes. AI’s recent ubiquitous presence in everyone’s life makes this technology an integral part of
daily activities that is often taken for granted. Most people treat AI tools as black boxes (i.e. devices
whose outputs are observable but whose internal details are unknown by users) which spit out
solutions to make problems go away. A reader of this book will quickly appreciate that there is no
magic behind the way AI operates. By understanding the mechanisms employed in many popular
AI techniques, correct solutions are easily obtained for complicated tasks. Through hands-on programming projects, the reader will be able to grasp how easily many otherwise unsolvable problems
can be handled using AI.
1.2 A Historical Perspective
The question of whether a human-created device can be intelligent goes back to the middle of the
twentieth century when the so-called Turing test was proposed to examine whether or not a
machine is capable of thinking [1, 2]. The Turing test assumes that a human interrogator asks questions to both another human and a computer through an impediment that prevents identification
of the responders. The proposition of the Turing test is that if the interrogator cannot distinguish
which answers come from whom, then the interrogated human and the machine have the same
intelligence. The Turing test has been used ever since to judge whether a computer can think. The
term artificial intelligence was coined in 1955, six years after the Turing test was introduced, as the science and engineering for making intelligent machines together with the Turing
test are considered as the beginning of modern AI [3].
The path from initial theoretical AI research to its realistic applications has progressed through
many successful breakthroughs. One of the prominent milestones for engineers was marked by
the introduction of the mobile robot Shaky in 1966 [4]. Shaky was controlled by so-called intelligent algorithms that analysed its surroundings to carry out a plan for an intended goal. On
another path, it has long been believed that being able to effectively play a chess game is a good
indicator for the effectiveness of AI. IBM developed a chess-playing computer called Deep Blue that
defeated the chess grandmaster Garry Kasparov, the chess champion of the world at that time, in a
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
2
1 Introduction
historic match in 1997 [5]. Recent developments in self-driving cars would not have been possible
without AI-based algorithms, which are essential for replicating the complex decision processes of
a human driver [6]. In 2015, car maker Tesla marked one of the greatest milestones by announcing
fully automated self-driving vehicles. By equipping its cars with AI software for interpreting and
understanding the visual world, Tesla showed that robust path planning and real-time decision
making were within reach [7]. These are only a few of the highlights marking recent developments
in AI. It is widely believed that the importance of AI will only grow in years to come.
1.3 Principles of AI
An AI can be classified by the areas for which it is designed to find solutions. Currently, these
areas include natural language processing, computer vision, automatic programming, robotics and
intelligent data retrieval systems. However, grouping AI by the computational concepts employed
for solving problems is a more accurate and better-suited classification for the readers of this book.
Searching is one of the computational methods that can be used for various AI tasks such
as reasoning (e.g. finding inference roles), planning (e.g. search through goals and sub-goals) and
moving (e.g. exploring a surrounding space by robots). Common AI tools to provide intelligent
decisions in situations with incomplete or uncertain information can involve engaging Bayesian
models, probabilistic algorithms and approaches derived from decision theory. Evolutionary algorithms are problem solving methods that attempt to replicate evolutionary operations of biological
organisms in searching through a solution space. They are often used in AI to learn from previous experiences and to find solutions in large and complex settings. Learning principles of AI can
be implemented using artificial neural networks and its variants, including convolutional neural networks and recurrent neural networks (RNNs), which allow
for a system to be trained on a known set of data and for subsequent prediction of outcomes
from new unknown events. Generative AI refers to a class of artificial intelligence systems that
are designed to create new content, such as text, images, audio or code, by learning patterns from
existing data. Unlike traditional AI, which often focuses on recognizing patterns or making predictions, generative AI models use those patterns to produce novel outputs that resemble the input
data but are not mere copies. These systems are typically built using techniques like deep learning
and neural networks, making them capable of generating realistic and creative content in various domains. Examples of generative AI methods include RNNs, long short term memory
(LSTM) [8] and transformers [9].
1.4 Applications That Are Impossible Without AI
Starting from the early days of the twenty-first century, AI has been used in countless applications ranging from solving a trivial task of suggesting emojis in text messages [10] to design of
flight-control software for commercial aeroplanes [11]. Some of the most prominent uses of AI are
in the areas of medical diagnosis, image processing, control of autonomous vehicles and prediction
of real-life events. In many similar tasks, an abundance of data (e.g. camera input, radar reading,
proximity sensor) has to be processed in real time, something at which AI is especially good.
Applications of AI can save lives, literally. In the healthcare field, high-risk patients can be
successfully monitored remotely with AI-based voice assistants. Automated physician assistance
systems guided by AI methods are capable of generating questions to best diagnose a patient [12]
1.5 Organization of This Book
and in finding an optimal and personalized treatment for them [13, 14]. During surgery, an
AI-based system can guide a surgeon’s scalpel so that soft tissue damage is minimized and the
resulting data collected during the procedures are quickly interpreted in clinical context [15].
AI can control an autonomous vehicle in places when a human operator cannot or does not
want to operate. Self-driving cars [16], unmanned ground vehicles, drones and underwater vehicles
[17–19] are only a few examples where the advantages of AI are directly visible.
Predicting the future may be one of the most powerful desires for many of us. Although AI cannot
find answers to important existential questions, it can be helpful in guessing the probabilities of
many future events [20]. For example, in an attempt to prepare for earthquakes, a combination of
evolutionary algorithms, swarm intelligence and artificial neural networks is being used to forecast
possible future seismic events. The natural language processing methods of AI can be employed in
economics to estimate upcoming changes in trading markets [21] and in judicial applications to
predict the outcome of a court ruling based on court proceedings [22].
1.5 Organization of This Book
This book is a beginner’s guide to exciting and modern applications of AI techniques. The reader
will be introduced to the basic concepts and principles of AI in order to develop skills readily
applicable to real engineering projects. Through step-by-step instructions, the reader will learn
how to implement typical AI tasks including control of autonomous drones, speech and character recognition, natural language processing, dietary menu planning, optimal selections for project
management tasks and maximizing profits in algorithmic stock trading. We believe that all readers,
regardless of their area of expertise, will enjoy an empowerment stemming from seeing relatively
simple AI procedures used to tackle otherwise formidable problems.
This book is written with a diverse spectrum of readers in mind: engineers, scientists, economists
and all backgrounds of students ranging from engineering to liberal arts. It welcomes all readers
who would like to enter the realm of AI-based problem solving as it has become an essential skill
for professionals in the twenty-first century.
In Chapter 2, the general concepts of an artificial neural network (ANN) architecture
and its training process are introduced. As one of the most popular tools in AI in handling tasks that
mimic events recorded in large amounts of data, ANN-based AI tools dominate the applications
that impact both our daily lives and scientific discoveries from astronomy to the social sciences.
However, when we focus on their operation, we find the process rather simple and straightforward:
a given set of sample data inputs are modified by a set of weights and activation functions so that
the outputs match the real data points measured in an experiment or event. We explore this process
and present a step-by-step analysis of ANN training.
In Chapter 3, we present two simplified but illustrative examples for optimized prediction using
ANNs. The first is the simple but popular ANN example for predicting an exam grade based on the
number of hours that a student studied and slept for this exam. The second example, inspired by the
data from bike rental companies operating in London, UK, involves constructing and training an
ANN to predict the number of bike rentals based on sample data such as the day of the week, time
of day, temperature and wind speed. We present sample Python and Matlab scripts implementing
these ANNs.
Chapter 4 explores the use of AI in the financial field. We introduce a simplistic ANN model
using popular financial technical indicators computed using daily stock market data as inputs to
3
4
1 Introduction
predict the price change for the following day as its output. Using existing libraries in Python and
Matlab, simplistic scripts are introduced for such an ANN.
Chapter 5 introduces the reader to AI applications in the life sciences. A simple case study is
presented, where medical data collected from Alzheimer’s disease (AD) patients are used to train an
ANN which exploits a possible relationship between expressions of AD-related genes and cognitive
exam scores. This ANN then can be deployed as a tool to predict personalized disease progression
for new patients.
Chapter 6 presents a natural language processing (NLP) application, where an ANN is
set up to explore a possible relationship between the tweets from a former president of the USA and
financial assets listed in public markets. ANN inputs include keywords extracted from tweets and
daily financial stock market data, and the output is the predicted price change for the following day.
In Chapter 7, we introduce the main concepts governing the operations and capabilities of convolutional neural networks (CNNs). These are ideal for identification (or classification)
of multi-dimensional inputs such as images and speech. We study the steps of training a CNN and
modification of weights for more accurate predictions. Chapters 8 and 9 present two very popular applications of CNNs, namely optical character recognition (OCR) and speech recognition,
respectively. For each chapter, simple but illustrative CNNs are implemented in Python and Matlab
scripts.
Recurrent neural networks (RNNs), which are the building blocks of language-related
applications, are introduced in Chapter 10. RNNs are fundamentally different to ANNs and CNNs
because RNNs can handle a sequence of inputs, where prior inputs have an impact on future
outputs. This capability of possessing a memory of previous inputs make RNNs a vital part of applications such as chatbots and machine translation. Training process for RNNs including forward and
back propagation operations is described in detail. Chapter 11 presents a simplistic chatbot architecture, which is then implemented in Python and Matlab scripts. This chatbot is trained with a
few hundred text message exchanges taken from the internet and demonstrates limited attempts
in generating responses for original user queries within the same dictionary of words.
In Chapter 12, the RNN with attention mechanism (RNN-Att) is introduced, which makes
use of historical inputs in an even more pronounced manner. For example, in natural language
translation, one has to wait until a sentence is complete before translating it to another language
since even the last words in a sentence may affect the translation significantly. We study the training
process, weight updating and reduction of error through forward and back propagation operation
in an RNN-Att. In Chapter 13, we present a simplistic RNN-Att architecture for machine translation
and its implementation in Python and Matlab scripts. The example translator is trained using a few
thousand German sentences and their English translations, and then demonstrates a remarkable
capacity to attempt translating new German sentences into English (using only the words in the
training dictionary).
In Chapters 14 to 18, we present prominent bio-inspired computation and evolutionary methods
in AI, which are typically applied to problems that are intractable or computationally too intensive
to be solved using classical methods. In Chapter 14, one of the most popular bio-inspired techniques, the genetic algorithm (GA), is introduced. With their relatively simple design, low
cost computational requirements and independence from software development platforms, GAs
are the perfect candidates for many real-life optimization problems in a wide variety of settings.
In the following chapters, we cover applications of GA, namely finding dietary menu selections
under nutritional restrictions (i.e. solving the ‘knapsack’ class of problems) in Chapter 15, flight
control of autonomous drones in Chapter 16 and finding optimum routes for travel (i.e. solving the
1.5 Organization of This Book
travelling salesman class of problems) in Chapter 17. In Chapter 18, we outline evolutionary computation methods including particle swarm optimization, differential evolution
and grammatical evolution.
By introducing them to such a wide spectrum of AI methods, we hope that readers will find
exciting and powerful new directions in their careers, as many of our students have done over the
years.
5
7
2
Artificial Neural Networks
2.1 Introduction
Biological nervous systems process information using nerve cells called neurons that transmit
messages to other cells using connections referred to as synapses. Information in the brain is
stored by strengthening synapses among a group of neurons through a reinforcement process
that repeatedly interchanges messages among these cells. Neurons operate in groups and react
to inputs applied to them by further propagating these inputs to other neurons only over the
strengthened synapses. No single neuron stores any particular piece of information, but a group
of neurons together with established synapses represent information in the brain [23].
Brain-inspired systems, called artificial neural networks (ANNs), intend to replicate
the way that humans learn. ANNs are built of nodes (i.e. artificial neurons) which are interconnected by directed links (i.e. synapses) representing a connection from the output of one artificial
neuron to the input of another. Each link in this system is associated with a weight, which alters
the information as it travels from one neuron to another. In an ANN, nodes are organized in layers responsible for processing information in sequence. Signals are processed starting from the
input layer, passing intermediate hidden layers until reaching the output layer, which generates
observable outputs. Figure 2.1 shows an example of an ANN consisting of an input, two hidden
and one output layers. Note that an engineer designing an ANN selects the number of nodes and
layers depending on the requirements of the problem to be solved. This design process requires
experience and specific domain knowledge for each application at hand.
2.2 Applications of ANNs
ANNs may be employed in solving many engineering problems ranging from cybersecurity [24]
to image processing [25]. In general, applications of ANNs fall into one of three categories:
classification, prediction or optimization problems. In classification problems,
a real-life observation is recognized as a particular category. One example is setting up an ANN to
classify activities of a credit card as either legitimate or fraudulent actions [26]. In this example, an
ANN can be built using legitimate actions on a given credit card based on the historic usage data
of this card (e.g. types of items bought, places visited, times of shopping, etc.). When an interaction
does not fit into the class of regular actions, it will be flagged as suspicious activity. Another
example of a classification problem involves recognizing handwritten characters as particular
letters or digits [27].
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
8
2 Artificial Neural Networks
Figure 2.1
An ANN with two hidden layers.
Prediction problems aim to estimate unknown outcomes based on historical data. For example,
an ANN can be built using past atmospheric data to predict future precipitation patterns and temperature levels [28]. Similarly, in stock market predictions, using historical data (e.g. recent market
behaviour, geopolitical situations and commodity prices), an ANN can be set up to predict whether
a given stock is expected to gain or lose value [29].
In optimization problems, valid outcomes, which often have to satisfy multiple and possibly
conflicting constraints, must be found in typically very large solution spaces. For example, finding
an optimal dose of a chemotherapy drug to be administered to a cancer patient based on personal
tumour growth data and overall health characteristics (e.g. heart condition, blood-sugar measurements, physiological indicators and others) may be addressed by an optimization-type ANN.
2.3 Components of ANNs
2.3.1 Neurons
In an ANN, each node (i.e. neuron) performs a calculation to determine if its input signals should
be forwarded to the next layer. This task is performed by so-called summation and activation
functions. First, a summation function adds the incoming signal strengths, and then an
activation function makes a decision whether or not to propagate them further to other layers of
the ANN. An example of summation and activation functions embedded in an artificial neuron are
shown in Figure 2.2, where inputs of xi and xj with weights wi and wj , respectively, are first summed
as a combined signal e. An activation function f (e) then determines if the resulting signal e should
be forwarded to the next layer. Most common activation functions are in the form of sigmoid
or rectilinear functions as discussed below. A sigmoid activation function transforms its
input into an output ranging between 0.0 and 1.0, which is useful when an output is probabilistic.
Figure 2.2 An ANN neuron employing summation
and activation functions.
2.3 Components of ANNs
A rectilinear activation function, on the other hand, generates an output signal equal to the input
signal strength for all positive signal values, and outputs zero for any input with non-positive
values.
2.3.2 Sigmoid Activation Function
The Sigmoid is convenient to efficiently calculate gradients used in ANN training (Figure 2.3).
Sigmoid activation limits neuron output within the interval of [0, 1]:
1
f (Z) =
.
(2.1)
1 + e−Z
The derivative of the sigmoid is given by
f ′ (Z) =
e−Z
.
(1 + e−Z )2
(2.2)
2.3.3 Rectilinear Activation Function
The rectilinear activation function (ReLU) and its derivative are shown in Figure 2.4. It is cheaper
to calculate gradients used in ANN training with rectilinear activation.
{
Z if Z > 0;
(2.3)
f (Z) =
0 otherwise.
The derivative of the rectilinear function is 1 for positive values of Z and 0 for negative values:
⎧1
if Z > 0;
⎪
f (Z) = ⎨0
if Z < 0;
⎪undefined if Z = 0.
⎩
′
(2.4)
(a)
Figure 2.3
(a) Sigmoid activation function, and (b) its derivative.
(a)
Figure 2.4
(b)
(b)
(a) Rectilinear activation function, and (b) its derivative.
9
10
2 Artificial Neural Networks
The rectilinear function prevents negative signals from propagating further in an ANN. However,
in the output layer of an ANN, eliminating negative values via a rectilinear function may be problematic if the outputs of the ANN are allowed to be negative. In these cases, rectilinear activation
functions should not be employed in the output layer.
2.3.4 Weights of Synapses
The value of a signal after it leaves an intermediate neuron is further altered by multiplying it by
the weight of the link over which it is travelling. Figure 2.5 shows an example ANN, where x1 and
x2 are the input signals to three nodes of a hidden layer. At the three hidden neurons, the weights of
the input synapses are multiplied by the input neuron values and aggregated at the hidden neurons
as follows:
e1 = (x1 ⋅ w1 ) + (x2 ⋅ w4 )
(2.5)
e2 = (x1 ⋅ w2 ) + (x2 ⋅ w5 )
e3 = (x1 ⋅ w3 ) + (x2 ⋅ w6 ).
An activation function f (.) is applied to the summed signals at each hidden neuron as follows:
a1 = f (e1 )
a2 = f (e2 )
a3 = f (e3 ),
where the a1 , a2 and a3 values are determined based on the activation function employed at the
neurons, as presented in Eqs. (2.1) and (2.3). Similarly, at the output layer, the output neuron has
three inputs which are aggregated as
e4 = (a1 ⋅ w7 ) + (a2 ⋅ w8 ) + (a3 ⋅ w9 ),
(2.6)
where the weights w7 , w8 and w9 are applied to the outputs of hidden neurons a1 , a2 and a3 , respectively. The output is obtained after applying an activation function to the summation shown in
Eq. (A.1):
ŷ = f (e4 ).
(2.7)
As discussed earlier, one has to be careful when employing a rectilinear activation function at an
output neuron as shown in Eq. (2.7), which allows for only positive output values, since negative
values may be important for certain applications.
Figure 2.5 An ANN with two inputs, one hidden layer and one output, whose aggregation and activation
functions are shown at each neuron.
2.4 Training an ANN
For the general case, if there are multiple hidden layers in an ANN, inputs and outputs of consecutive hidden layers are computed in the same manner explained here. The output layer may
contain more than one neuron as an ANN may be designed for multiple outputs depending on the
application. For example, an ANN to guide a drone may generate three outputs (speed, direction
and altitude) for a flight controller using multiple inputs such as current position, speed, altitude,
flight plan and information obstacle sensors.
2.4 Training an ANN
In order for an ANN to generate predictions, it first has to be trained using existing sample data
of input and output values, in a manner similar to traditional trial-and-error approaches. During
training, weights associated with the synapses (e.g. w1 to w8 in Figure 2.5) are adjusted so that the
difference between the outputs generated by the ANN and the actual output values of the existing
data are as small as possible. This difference is called the error or loss. The goal of training is
to minimize (or reduce as much as possible) the loss by adjusting the weight values.
Let us now illustrate the training process using the example ANN given in Figure 2.5. Suppose
we have the following data points observed in field as the inputs and output of an event for which
an ANN is to be trained:
input 1: x1 = 3
input 2: x2 = 5
output: y = 75.
Suppose that current initial values of the ANN weights are
w1 = w2 = w3 = 1.0
w4 = w5 = w6 = 0.5
w7 = w8 = w9 = 4.0.
Using Eqn. (A.1) we have
e1 = (3 ⋅ 1) + (5 ⋅ 0.5) = 5.5
e2 = (3 ⋅ 1) + (5 ⋅ 0.5) = 5.5
e3 = (3 ⋅ 1) + (5 ⋅ 0.5) = 5.5.
Using a rectilinear activation function, from Eq. (2.6), we have
a1 = 5.5
a2 = 5.5
a3 = 5.5.
From Eq. (A.1), we then have
e4 = (5.5 ⋅ 4) + (5.5 ⋅ 4) + (5.5 ⋅ 4) = 66.
Using a rectilinear activation function, from Eq. (2.7), we have
ŷ = 66.
The error is defined as the difference between the ANN-generated output and the real output
value from the training data. For this example, the error is
error = y − ŷ = 75 − 66 = 9.
(2.8)
11
12
2 Artificial Neural Networks
For real-life applications, different methods of error computation are possible, for example using
squares of differences or logarithmic values [30]. For the sake of argument, suppose the weights are
adjusted through the ANN training process such that the new weight values are now as follows:
w1 = w2 = w3 = 1.0
w4 = w5 = w6 = 0.8
w7 = w8 = w9 = 3.5.
Using these weight values with the sample data, we have
e1 = (3 ⋅ 1) + (5 ⋅ 0.8) = 7
e2 = (3 ⋅ 1) + (5 ⋅ 0.8) = 7
e3 = (3 ⋅ 1) + (5 ⋅ 0.8) = 7,
which yields
e4 = (7 ⋅ 3.5) + (7 ⋅ 3.5) + (7 ⋅ 3.5) = 73.5.
In this case, the difference between the real output value of y and the output generated by the
ANN of ŷ (i.e. the error) becomes
error = y − ŷ = 75 − 73.5 = 1.5.
(2.9)
We can clearly see from this simple enumeration shown in Eqs. (2.8) and (2.9) that it is possible
to reduce the error by adjusting the values of the synapse weights, which is the fundamental idea
behind the ANN training process. An ANN is trained by using the input and output data (also
referred to as historic, sample or field data) to assign proper weight values to links. The training
process can be summarized by the following steps:
To start, randomly chosen values are assigned to the link weights.
Using these random values, the output values are calculated by multiplying the inputs and
weights and applying the activation functions at each neuron of the ANN.
● The steps up to this point are referred to as forward propagation (see Section 2.5 below).
● The outputs are then compared with the expected values from the sample data. The difference
between the ANN-generated and real output values is the error generated by the ANN.
● After forward propagation and calculation of loss, a back propagation is performed based
on partial derivations of loss with respect to ANN weights at each layer (see Section 2.6 below).
● Based on the partial derivations, the weight values are updated.
● Using the new weights, consecutive forward and back propagation steps (called epochs) are
performed as above.
● End of training is determined by the designers based on various criteria, including performing a
pre-determined number of epochs, the loss value reaching a desired low value or lack of progress
from epoch to epoch in terms of loss reduction.
●
●
2.5 Forward Propagation
Let us now explore the forward propagation step of the ANN training process. For the derivations
in this section, without loss of generality, we will use a simple ANN with one hidden layer as given
in Figure 2.6 (inspired by the example in [31]). For the general case, with ANNs having multiple inputs, outputs and hidden layers, the derivations are valid with weight matrix dimensions
2.6 Back Propagation
Figure 2.6
in [31]).
Example ANN with two inputs, one hidden layer and one output (Adapted from the example
increased accordingly. Recall that each synapse of an ANN has an independent weight, which is
multiplied by the output of the starting neuron.
2.5.1 Forward Propagation from Input to Hidden Layer
For forward propagation from input to hidden layer of the ANN shown in Figure 2.6, we have
(2)
(1)
Z(11)
= x1 ⋅ w(1)
11 + x2 ⋅ w21
(2)
Z(12)
= x1 ⋅ w(1)
+ x2 ⋅ w(1)
12
22
(2)
Z(13)
= x1 ⋅ w(1)
+ x2 ⋅ w(1)
.
13
23
Using the definition of sigmoid activation given in Eq. (2.1), we have the following outputs from
the hidden layer neurons:
1
(2)
)=
a2(11) = f (Z(11)
(1)
−(x1 ⋅w(1)
11 +x2 ⋅w21 ) )2
(1 + e
1
(2)
a2(12) = f (Z(12)
)=
−(x1 ⋅w(1)
+x2 ⋅w(1)
)
12
22 )2
(1 + e
1
(2)
a2(13) = f (Z(13)
)=
.
−(x1 ⋅w(1)
+x2 ⋅w(1)
)
13
23 )2
(1 + e
At the output layer neuron, aggregation of the three input synapses yields
(2)
(2)
(2)
(2)
(2)
Z (3) = a(2)
11 ⋅ w1 + a12 ⋅ w2 + a13 ⋅ w3 .
Finally, at the output layer, using a sigmoid activation function generates ŷ as
1
1
ŷ = f (Z (3) ) =
=
.
−(a(2)
⋅w(2)
+a(2)
⋅w(2)
+a(2)
⋅w(2)
)
1 + e−Z
11
1
12
2
13
3
1+e
2.6 Back Propagation
2.6.1 Back Propagation for a Neuron
Once forward propagation is performed, the error must be back propagated starting from output layer all the way to the input layer so that the synapse weights can be updated based on the
13
14
2 Artificial Neural Networks
Figure 2.7 A neuron with two inputs xa and xb
incoming over synapses using weights wa and
wb generating output of ŷ .
error values. Let us explore how the back-propagation process works for a given neuron. Without
loss of generality, suppose there are two inputs, namely xa and xb , over two incoming links with
weights wa and wb , as shown in Figure 2.7. Let the output of this neuron be ŷ = f (e), where e is the
aggregation and is defined as e = xa wa + xb wb . To minimize the error on ŷ , one has to compute its
derivation with respect to e. For a rectilinear activation function, we then have
𝜕 ŷ
.
𝜕e
Partial derivatives of ŷ with respect to wa and wb are as follows:
f ′ (e) =
𝜕(xa wa + xb wb )
𝜕 ŷ
=
= xa
𝜕wa
𝜕wa
𝜕(xa wa + xb wb )
𝜕 ŷ
=
= xb .
𝜕wb
𝜕wb
For the real value of y at this neuron, error E is defined as
1
E = (y − ŷ )2 .
2
The partial derivative of E with respect to wa is
𝜕 ŷ
𝜕 ŷ 𝜕e
𝜕E
= −(y − ŷ )
= −(y − ŷ )
.
𝜕wa
𝜕wa
𝜕e 𝜕wa
Let us define 𝛿 = −(y − ŷ )f ′ (e) and plug it into the above error equation:
𝜕e
𝜕E
=𝛿
= 𝛿xi .
𝜕wa
𝜕wa
(2.10)
Similarly, the partial derivative of E with respect to wb is
𝜕 ŷ
𝜕 ŷ 𝜕e
𝜕E
= −(y − ŷ )
= −(y − ŷ )
.
𝜕wb
𝜕wb
𝜕e 𝜕wb
Again, using 𝛿 = −(y − ŷ )f ′ (e) we have
𝜕e
𝜕E
=𝛿
= 𝛿xj .
𝜕wb
𝜕wb
(2.11)
Using the values from Eqs. (2.10) and (2.11), updating the weights for a learning rate of 𝛼 will give
wa = w a − 𝛼
𝜕E
𝜕wa
wb = wb − 𝛼
𝜕E
,
𝜕wb
(2.12)
where 𝛼 is a real number specifying the modification rate of weights during training. We will see in
the reminder of this chapter that synapse weights will be modified in a similar fashion to Eq. (2.12).
2.6 Back Propagation
2.6.2 Back Propagation – from Output to Hidden Layer
Let us now derive back-propagation equations for the ANN shown in Figure 2.6. The error at the
output layer is defined as half of the square of the difference between the real and the calculated
output values, namely y and ŷ , respectively:
∑1
∑1
J=
(y − ŷ )2 =
(y − f (f (XW (1) )W (2) )2 ).
2
2
To reduce the error, J has to be minimized, which means that its partial derivatives with respect
to W (1) and W (2) must be equal to 0.
∑
𝜕
𝜕 ∑1
(y − ŷ )2 =
(y − ŷ ).
(y − f (f (XW (1) )W (2) )2 )
𝜕W
2
𝜕W
Since the link weights between output and hidden layers are represented as W (2) as shown in
Figure 2.6, let us first calculate the partial derivative of J with respect to W (2) :
𝜕 ∑1
𝜕 ∑
𝜕J
(y − ŷ )2 = (y − ŷ )
=
(y − ŷ )
(2)
(2)
2
𝜕W
𝜕W
𝜕W (2)
∑
𝜕
ŷ .
= −(y − ŷ )
𝜕W (2)
Since ŷ = f (Z(3)), we have
∑
∑
𝜕 ŷ
𝜕 ŷ 𝜕Z (3)
𝜕J
̂
̂
=
−(y
−
y
)
=
−(y
−
y
)
.
𝜕W (2)
𝜕W (2)
𝜕Z (3) 𝜕W (2)
Given that Z (3) = a(2) ⋅ W (2) , we get
𝜕Z (3)
= a(2) .
𝜕W (2)
For the sigmoid activation function, we have
1
f (Z (3) ) = ŷ =
,
1 + e−Z(3)
which means that
∑
𝜕 ŷ
e−Z(3)
= f ′ (Z (3) ) =
.
(3)
𝜕Z
(1 + e−Z(3) )2
We now have
𝜕J
= −(y − ŷ )f ′ (Z (3) )a(2) .
𝜕W (2)
Let us define the back-propagated error, called 𝛿 (3) , as follows:
𝛿 (3) = −(y − ŷ ) ⋅ f ′ (Z (3) ),
(2.13)
where −(y − ŷ ) and f ′ (Z (3) ) are multiplied element by element. We now have
𝜕J
= 𝛿 (3) a(2) .
𝜕W (2)
We should note that back propagation is based on gradient descent, which is prone to getting
stuck at local extreme points. Since f ′ (Z (3) ), y and ŷ have the same dimensions (i.e. the number of
samples by the number of outputs), 𝛿 (3) will also have the same dimensions. However, to perform
a matrix multiplication for 𝛿 (3) and a(2) , we must transpose a(2) , which yields
𝜕J
= a(2)† 𝛿 (3) ,
𝜕W (2)
where † represents the matrix transposition operation.
(2.14)
15
16
2 Artificial Neural Networks
2.6.3 Back Propagation – from Hidden Layer to Input
For back propagation from hidden to input layer, we need to calculate the partial derivative of the
error J with respect to link weights of W (1) as shown in Figure 2.6. Hence,
𝜕 ∑
𝜕 ∑1
𝜕 ∑
𝜕J
2
̂
̂
̂
̂
(y
−
y
)
=
−(y
−
y
)
ŷ .
(y
−
y
)
=
=
(y
−
y
)
2
𝜕W (1)
𝜕W (1)
𝜕W (1)
𝜕W (1)
Since ŷ = f (Z(3)), we have
∑
∑
𝜕 ŷ
𝜕 ŷ 𝜕Z (3)
𝜕J
̂
̂
=
−(y
−
y
)
=
−(y
−
y
)
.
𝜕W (1)
𝜕W (1)
𝜕Z (3) 𝜕W (1)
We know that
𝜕Z (3) 𝜕a(2)
𝜕Z (3)
=
.
(1)
𝜕W
𝜕a(2) 𝜕W (1)
We then obtain
𝜕Z (3) 𝜕a(2)
𝜕J
= −(y − ŷ )f ′ (Z (3) ) (2)
.
(1)
𝜕W
𝜕a 𝜕W (1)
Given that Z(3) = a(2) ⋅ W (2) , we get
𝜕Z(3)
= W (2)† .
𝜕a(2)
Recall from Eq. (2.13) that we defined 𝛿 (3) as −(y − ŷ )f ′ (Z(3)). We know that
𝜕a(2) 𝜕Z (2)
𝜕a(2)
=
.
(1)
𝜕W
𝜕Z (2) 𝜕W (1)
Therefore, we now have
(2) 𝜕Z (2)
𝜕J
(3)
(2)† 𝜕a
=
𝛿
W
.
𝜕W (1)
𝜕Z (2) 𝜕W (1)
Recall from Figure 2.6 that a(2) = f (Z (2) ); then we have
𝜕a(2)
= f ′ (Z (2) ).
𝜕Z (2)
Also recall from Figure 2.6 that Z (2) = XW (1) ; we now have
𝜕Z (2)
= X.
𝜕W (1)
This yields
𝜕J
= 𝛿 (3) W (2)† f ′ (Z (2) )X.
𝜕W (1)
As before, the dimensions need to be adjusted as
𝜕J
= X † 𝛿 (3) W (2)† f ′ (Z (2) ).
𝜕W (1)
Let us define 𝛿 2 as follows:
𝛿 2 = 𝛿 (3) W (2)† ⋅ f ′ (Z (2) ),
(2.15)
where f ′ (Z (2) ) needs to be transposed and multiplied by W (2)† element by element. Then we have
𝜕J
= X † 𝛿 (2) .
𝜕W (1)
(2.16)
2.8 ANN with Input Bias
2.7 Updating Weights
During training of an ANN, after each back propagation, the weights are updated such that the
error is reduced after each iteration, which is based on the modifications of the gradients modified
through the hidden layers as described above.
Let 𝛼 be defined as the learning rate, a real number that specifies the amount of modification chosen in updating the weights during training. Using the derivatives in Eqs. (2.14) and (2.16),
the weights will be updated at each iteration as follows:
𝜕J
𝜕W (1)
𝜕J
(2)
Wnew
= W (2) − 𝛼
,
𝜕W (2)
(1)
= W (1) − 𝛼
Wnew
(2.17)
where the weights on the left-hand side with subscript new are the updated values at that iteration,
whereas the right-hand side of the equations have the current weight values. After updating the
weights, a new forward and back propagation are performed. At each iteration the weights are
updated using the new gradients with Eq. (2.17). Successive updatings of weights are expected to
reduce the error in ANN outputs. Training continues until a satisfactorily small amount of error is
obtained or a pre-determined number of iterations are performed.
2.8 ANN with Input Bias
In real-life problems, adjusting the weights to minimize output errors requires that a bias is
employed at every layer of the ANN, as shown in Figure 2.8. Bias values, similar to synapse
weights, are first randomly initialized. During training, they are propagated forward and back and
updated at each iteration of training. Step by step derivation of updating bias and weight values
during the training process using input bias is presented in Appendix B.
Figure 2.8
Example ANN with two inputs with bias, one hidden layer and one output.
17
18
2 Artificial Neural Networks
Figure 2.9
A simple algorithm describing the ANN training process.
2.9 A Simple Algorithm for ANN Training
A simple algorithm outlining the ANN training process is shown in Figure 2.9. The algorithm first
defines the structure of the ANN and the number of epochs (i.e. forward and back propagations) to
be used for training. Next, it initializes random weights and biases for all layers. For each epoch, it
performs forward propagation to yield a preliminary output of the ANN and its associated error or
cost. After forward propagation, it uses back propagation to calculate the error and the gradients
to adjust the weight and bias values. This process is repeated until a maximum number of epochs
have been reached.
Once the weight values are decided, the trained ANN can then be used to generate predictions
for new inputs. Different ANN training processes include:
Supervised training: inputs are mapped to a set of known output signals [32].
● Unsupervised training: this uses inputs to form data categorization [33].
● Reinforcement training: this continuously improves an ANN based on new input data
evaluated by a fitness function [34].
●
2.10 Computational Complexity of ANN Training
Computational complexity of training includes all the operations performed in Sections 2.5, 2.6
and 2.7. They are repeated at each epoch for each of the data samples available in training. As can
be seen in Figure 2.5, in the first hidden layer, each neuron performs a summation proportional
to the number of inputs multiplied by the number of links: since the hidden layer has 6 incoming
2.11 Normalization of ANN Inputs and Outputs
links to 3 neurons, there are 6 sets of multiplications and 3 computations of the activation function.
For the single output layer, there are 3 incoming links, there are 3 multiplications and 3 computations of activation function. Let the number of links and neurons in the ith layer be |W (i) | and
|S(i) |, respectively. For the general case, for forward propagation, the number of multiplications
and activation function computations for one input–output sample are
1
costM
=
H
∑
|W (i) ||S(i) |cm
costA1 =
i=1
H
∑
|W (i) ||S(i) |ca ,
i=1
where cm and ca are the computational costs for multiplication and activation function computation, respectively. For the sake of simplicity, suppose back-propagation cost is comparable to
forward propagation for a given iteration. If there are |X| samples, the computational cost of training for one epoch is
(H
)
∑
1
(i)
(i)
costANN = 2|X|
|W ||S |(cm + ca ) .
i=1
For N forward- and back-propagation iterations, the training cost becomes
(H
)
∑
N
(i)
(i)
costANN = 2N|X|
|W ||S |(cm + ca ) .
(2.18)
i=1
Using  notation to indicate the order of magnitude for computational complexity [35], the cost
for training a fully-connected ANN (i.e. all neurons are connected to all others in consecutive hidden layers), we have
N
costANN
= (N |X| H |W| |S|)(cm + ⋅ca ).
(2.19)
Consider a typical numerical application for a fully-connected ANN with H = 32 layers, each
with S = 32 neurons, using X = 105 data samples over N = 104 iterations. Using Eq. 2.18, the
computational cost for training such an ANN is
)
( 32
∑
104
4
5
3
10 (cm + ca ) = (1013 )(cm + ca ).
costANN = 2 ⋅ 10 ⋅ 10
i=1
Since for realistic applications, training may require hundreds or thousands of iterations using
tens of thousands of samples over ANNs with tens of hidden layers each of which has tens of neurons, computational complexity may quickly become infeasibly large if design engineers are not
cognizant of such costs.
2.11 Normalization of ANN Inputs and Outputs
For applications where there are numerical imbalances between input and output values, it is possible to experience computational inaccuracies due to rounding, truncation and similar types of
approximations. To alleviate such numerical problems, inputs and outputs are typically scaled
down (i.e. normalized) to be within the range of relatively small integers. An example of normalization is as follows:
Xi − min(X)
Yi − min(Y )
, Yi =
,
Xi =
max(X) − min(X)
max(Y ) − min(Y )
19
20
2 Artificial Neural Networks
where Xi and X i are the values of the ith input before and after normalization, respectively. Similarly, Yi and Y i are the ith output values before and after normalization, respectively. Depending
on the application, inputs are normalized separately over multiple samples. Similarly, values of a
single output over multiple samples are normalized. If a group of inputs are at the same level of
importance, they may be normalized together. For example, if the input values are positive, we can
use min(X) = 0. Similarly, if the output range is [0, 100], we have min(Y ) = 0 and max(Y ) = 100.
Normalization is a sound practice since it tries to prevent precision issues in calculations and helps
accelerate convergence to better solutions [36].
2.12 Concluding Remarks
In this chapter, general concepts of ANN architecture and training process are outlined. ANNs are
one of the most popular tools in AI for solving problems that are based on large amounts of data. In
an ANN, a given set of inputs are modified by a set of weights and activation functions to generate
outputs that match the real data points measured in an experiment or an event. There are a class
of problems that are suitable to be solved by ANNs. For example, classification problems
and image identification tasks are excellent candidates for ANNs. In such applications, there are
typically a large number of inputs, and their respective outputs are available (typically sample sizes
range from tens to hundreds of thousands or higher). A process called training aims to find the
weight values such that the difference between real and calculated values of outputs are minimized
(or they are as small as possible within the limits defined for the process).
There have been several different types of ANNs developed in recent years. Convolutional neural
networks (CNNs), recurrent neural networks (RNNs), large language models (LLMs) and transformers are among the most popular ones. In an RNN (See Chapters 10 and 11), values from the
output layer are sent back to the hidden layers to be used in follow-up processing of the signal.
Using the attention mechanism in RNNs, machine translation tasks become feasible, as discussed
in Chapters 12 and 13. A good survey for language models can be found in [37].
Modelling challenging real-life problems to be solved by ANNs requires experience and the analytical skill set of engineering disciplines. Computational complexity of training grows by the product of the number of neurons and the number of hidden layers. Therefore, one has to be cognizant
of computational cost of training when designing ANN architectures for real-life applications.
2.13 Exercises for Chapter 2
1) How would the performance of the ANN given in Figure 2.6 be impacted if there were two
hidden layers?
2) How would the performance of the ANN given in Figure 2.6 be impacted if the learning rate of 𝛼
were increased tenfold in terms of converging to an acceptable solution by reducing the output
error?
3) How would the performance of the ANN given in Figure 2.6 be impacted if the learning rate of 𝛼
were decreased tenfold in terms of converging to an acceptable solution by reducing the output
error? Compare your answer with Exercise 2.
4) Consider the ANN given in Figure 2.6. Suppose one doubles the number of neurons. How will
the ANN performance be impacted in terms of converging to an acceptable solution by reducing
the output error?
2.13 Exercises for Chapter 2
5) Consider the ANN given in Figure 2.6. How will its performance be impacted in terms of converging to an acceptable solution by reducing the output error if one doubles the number of
neurons and the number of hidden layers? Compare your answer with Exercise 4.
6) How will the performance of the ANN given in Figure 2.6 be impacted in terms of converging to
an acceptable solution by reducing the output error if one doubles the number of neurons and
the number of hidden layers in addition to the modification suggested in Exercise 2?
21
23
3
ANNs for Optimized Prediction
3.1 Introduction
ANNs can be effectively used for prediction of outcomes based on historical data if they are properly
designed and effectively trained with sufficiently large numbers of samples. To be able train the
weights of an ANN, a set of sample input and output data collected from the field or observed
during an experiment are needed, as described in Chapter 2. Depending on the application, the
number of samples typically ranges from thousands to hundreds of millions or even more. Input
values of data samples are applied to the neurons of a hidden layer after being multiplied by the
link weights. Following modification by summation and activation functions at each neuron, input
signals propagate to the next hidden layer and finally to the ANN output layer. An error value is
computed based on the difference between the real and calculated output values. These steps are
called a forward propagation. Then, as part of the error minimization (or at least reduction) process,
a back propagation is performed by computing partial derivatives of the error with respect to the
link weights starting at output layer and working back to the input layer, as described in Chapter 2.
Consecutive forward and back propagation iterations (called epochs) over ANN layers reduces
the error until either a satisfactorily small value is obtained, a limit for the number of epochs is
reached or another stopping criterion is met. As mentioned earlier, for an ANN to be properly
trained, large sets of data are needed. The larger the sample size, the more accurate the predictions are expected to be from an ANN. Training with fewer data samples tends to over-fit
link weights to satisfy the input and output values, which causes inaccurate predictions when
inputs with different value ranges are applied to the ANN. The intention of using a large number of samples is that they hopefully have a wide range of input and output values to cover the
operation space adequately – having many samples with similar values will not help to train an
effective ANN.
In this chapter, we provide two examples of arguably one of the most popular applications of
ANNs: outcome prediction based on historical data. The first example, presented in Section 3.5,
involves designing an ANN to be used to predict the grade from an exam based on two inputs,
namely, the hours of sleep and study spent by a student in preparation for the exam (the example is
taken from [31]). For this simplistic but very relatable example, we present step-by-step descriptions
of forward- and back-propagation operations.
The second example, in Section 3.6, constructs and trains an ANN to predict the number of rental
bikes needed for a future date based on several inputs, including the day of the week, the air temperature and the wind speed (taken from [38]). This case study is inspired by data obtained for bike
rentals in London, UK.
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
24
3 ANNs for Optimized Prediction
Real-life versions of these example ANNs can be used to predict the outcomes for new situations.
For example, a student can predict the expected number of hours to sleep and study for an incoming
exam in order to receive a grade over 90, which may require schedule adjustments accordingly.
Similarly, a bike rental company can predict how many bikes will be needed at the weekend and
perhaps hire extra help to manage the rentals as needed.
3.2 Selection of ANN Inputs
When constructing an ANN for a given application, one of the most important decisions is the
selection of ANN inputs. For example, one may consider using the data collected from experiments
to be considered as ANN inputs (e.g. the number of hours that a plant is exposed to the sun, the
amount and frequency of water given to the plant and so on). For some applications, measurements
in the field utilized as ANN inputs may be effective in generating acceptable predictions. However,
applying observed data directly as ANN inputs may not be preferable for certain cases, especially if
the number of inputs is infeasibly large or is not directly related to outputs (e.g. it may be irrelevant
to use names of the weekdays for a plant growth application). ANN inputs may also be derivatives of
data collected from field experiments or observations of events, typically statistical or mathematical
transformations of the observed data. For example, it may be preferable to use the average, highest
or lowest daily temperatures as an ANN input instead of hourly measurements recorded during
experiments. In another application, the moving average of a financial stock calculated over the
last several weeks may be preferable to directly using its selling price obtained from stock exchange
records. It is also possible to combine directly observed measurements with the values derived from
them. For example, average air temperature together with the day of a week could be relevant inputs
for an ANN designed for predicting daily bus repairs in a transportation company. If a weekday
is Friday, Saturday or Monday may be irrelevant to a plant watering study but has considerable
significance for bus schedules. In practice, designers employ whatever available data they deem to
be suitable for their purposes and decide which combination of inputs will train an ANN for the
most accurate prediction. Selection of proper inputs is an ongoing process and may take significant
effort in considering different inputs sets until a satisfactory set is defined.
For the two examples studied in this chapter, to keep them as simple as possible, all inputs are
directly measured parameters in the respective fields: number of hours a student sleeps and studies
for the first example, and various measurements kept by the bike rental company, such as wind
speed, temperature, time of day and similar measurements.
3.3 Selection of ANN Outputs
In addition to selection of its inputs, successful deployment of an ANN also requires that the outputs
are chosen in such a way that the ANN generates the most useful, accurate and reliable predictions.
Similar to the selection of inputs, outputs can be assigned either as directly measured observations for an experiment or as derived values based on measurements. For example, for an ANN
constructed to predict growth of a cancerous tumour, possible output selections may include the
tumour size measurements or the growth rate of the tumour. For the latter case, the tumour size of
a patient would need to be measured over a period of time and a growth (or shrinkage) rate computed based on the measurements. In other examples of ANNs, outputs can represent classification
decisions. For example, for an ANN whose input is the text of an email, a possible output may be
3.5 Case Study 1: Sleep-Study Example
the decision as to whether the email is genuine or spam. For this case, there may be two outputs
(e.g. labelled as genuine and spam), only one of which will be true, or a single output whose value
indicates the ANN decision (e.g. 1 is a genuine email and 0 is spam). For a given ANN application,
selection of suitable outputs typically requires experience and domain knowledge.
In the two case studies presented in this chapter, for the sake of simplicity, directly observable
outputs are selected: expected exam grade for the so-called sleep-study ANN and the number of
rentals for the London bike sharing ANN.
3.4 Construction of Hidden Layers
Another important design component is the selection of the number of hidden layers and the
number of neurons in each layer. There is no straightforward formula that dictates the rules for
these decisions. Similar to other ANN design parameters, a knowledge- and experience-guided
trial-and-error approach seems to be a reasonable path in constructing an ANN architecture that
consistently generates reliable and accurate solutions for a given application. For instance, image
processing problems may require ANNs with many hidden layers to detect various features in an
input image (e.g. tumours, nodules or other anomalies), whereas financial applications may be
more successful with ANN architectures employing smaller numbers of hidden layers but each
having a large number of neurons (e.g. for detection of financial trends without over-fitting the
link weights over many hidden layers).
The time required to train an ANN is an important performance metric, especially for cases
where they need to be trained often (e.g. a few times per day or more frequently). As discussed
in Section 2.10 in Chapter 2, training time is directly proportional to the number of hidden layers
and the number of neurons in each layer, together with the number of connections between consecutive layers. As implied by Eqs. (2.18) and (2.19), increasing the number of hidden layers will
exponentially increase the computational cost of training, which is a serious issue to be handled by
the ANN designer if there are a large number of input/output samples to be used for training.
Another important observation from Eqs. (2.18) and (2.19) is the selection of number of epochs to
be used in training, where link weights and biases are adjusted based on error values observed at the
outputs. It is clear that for ANNs requiring large sets of data samples (e.g. hundreds of thousands
or millions), one cannot arbitrarily increase the number of hidden layers, neurons and epochs for
applications where the ANN will be re-trained often (e.g. hourly, daily or weekly). If an ANN for
reliable predictions of stock movements for the next day requires a week to train, the ANN outputs
will not be meaningful to its users no matter how accurate they may be. These trade-offs are an
essential part of the ANN design process, especially for critically important applications, such as
medical, military, live traffic, power, water supply or nuclear control tasks.
3.5 Case Study 1: Sleep-Study Example
In our first example (from [31]), which we will refer to as the sleep-study example, an ANN
is set up to predict the exam grade of a student based on two measurements, namely the number
of hours that the student spent studying and the number of hours slept during exam preparation.
ANN inputs are selected as the direct measurements of these two parameters. ANN output is the
expected grade for that exam. For the sake of simplicity, suppose that the training data for this
simple example contains the samples shown in Figure 3.1.
25
26
3 ANNs for Optimized Prediction
Figure 3.1 Data samples for the sleep-study example: for each sample, the inputs are the number of hours
that a student slept and studied for an exam, and the output is the grade received for that exam.
Figure 3.2
A fully-connected ANN for the sleep-study example using one hidden layer with three neurons.
Let us select a fully-connected ANN with only one hidden layer containing three neurons
(Figure 3.2). As described in Chapter 2, the ANN is first trained on historical data, which consists
of the number of hours spent on sleeping and studying with the outcomes of the grades received
for those exams. At the start of training, link weights and bias are assigned randomly selected
values. After one forward propagation from the input to the output layers through the hidden
layer, a prediction for output is obtained, and its error is computed as the difference between the
real and calculated values of the grades. Through a back propagation, partial derivations of the
error with respect to the link weights and bias are computed such that the weight matrix and bias
values are adjusted to reduce the error. In our Python and Matlab implementations of this ANN,
presented in Sections 3.5.9 and 3.5.10, respectively, forward and back propagations are repeated
2 000 times.
After training is complete, the ANN can be used to predict scores for new exams. When anticipated number of hours for sleep and study are entered to the ANN for an incoming exam, it
generates a grade prediction. If the grade is satisfactory, the student may follow that schedule for
sleep and study. However, if the grade is too low, the student has a chance to experiment with the
ANN by using different sets of input values to find optimal sleep and study hours yielding a satisfactory grade. Of course, we recommend that students not spend too much time experimenting
with an ANN but instead focus on learning the course materials as much as possible, regardless of
the grade desired.
3.5.1 Using Matrices for ANN Training
Let us now consider representing multiple samples used during training as matrices. The input
values shown in Figure 3.1 can be presented as a matrix, called X, with two columns and three rows
3.5 Case Study 1: Sleep-Study Example
representing the number of different inputs and the number of samples, respectively. Similarly, the
sample output values can be arranged to form a vector called Y :
⎡ 3 5⎤
X = ⎢ 5 1⎥
⎢
⎥
⎣10 2⎦
⎡75⎤
Y = ⎢82⎥ .
⎢ ⎥
⎣93⎦
Once the inputs and outputs are represented as matrices, instead of handling each sample separately, multiple samples can be handled all at once in training using matrix operations. Although
computational cost is not reduced, using matrices make script implementation of ANNs considerably more convenient compared to additions and multiplications of each individual sample
separately.
For the sake of simplicity and without loss of generality, let us use the samples given in Figure 3.1
in determining the matrix elements below for the weights at various stages of the simple ANN. As
mentioned above, for input samples, matrix dimensions are the number of samples by the number
ANN inputs. For the samples shown in Figure 3.2, the matrix for the inputs will have dimensions
of (3 × 2):
⎡x11 x12 ⎤
X = ⎢x21 x22 ⎥ .
⎢
⎥
⎣x31 x32 ⎦
Matrix dimensions for the weights between the input and hidden layers is the number of ANN
inputs by the number of neurons in the hidden layer. For the ANN in Figure 3.2, the matrix for the
weights will be a (2 × 3) matrix:
⎡w(1) w(1) w(1) ⎤
11
12
13
(1)
⎥.
W = ⎢
⎢w(1) w(1) w(1) ⎥
⎣ 21 22 23 ⎦
For the outputs of hidden layer, we then have
⎡z(2) z(2) z(2) ⎤
⎡x11 x12 ⎤ ⎡w(1) w(1) w(1) ⎤ ⎢ 11 12 13 ⎥
11
12
13
⎥ = ⎢z(2) z(2) z(2) ⎥ .
Z (2) = X ⋅ W (1) = ⎢x21 x22 ⎥ ⋅ ⎢
⎥ ⎢w(1) w(1) w(1) ⎥ ⎢ 11 12 13 ⎥
⎢
⎣x31 x32 ⎦ ⎣ 21 22 23 ⎦ ⎢ (2) (2) (2) ⎥
⎣z11 z12 z13 ⎦
(3.1)
For the general case, the matrix for weights between two hidden layers will the number of neurons of current hidden layer by the number of neurons of the next hidden layer. Since the example
in Figure 3.2 has only one hidden layer, the number of neurons between the hidden and output
layers will be the number of hidden neurons by the number of ANN outputs, which is a (3 × 1)
matrix as follows:
(2)
⎡w1 ⎤
⎥
⎢
W (2) = ⎢w(2)
.
2 ⎥
⎢ (2) ⎥
⎣w3 ⎦
(3.2)
Therefore, the matrix representing the outputs of the ANN in Figure 3.2, which is the matrix
resulting from multiplication of matrices in Eqs. (3.1) and (3.2), has the dimensions of the number
of samples by the number of outputs:
(2) (2) (2)
(2)
(3)
⎡z11 z12 z13 ⎤ ⎡w1 ⎤ ⎡z1 ⎤
⎢
⎢
⎢
⎥
⎥
⎥
(2) (2) (2)
= ⎢z2(3) ⎥ .
Z (3) = a(2) ⋅ W (2) = ⎢z11
z12 z13 ⎥ ⋅ ⎢w(2)
2 ⎥
⎢ (2) (2) (2) ⎥ ⎢ (2) ⎥ ⎢ (3) ⎥
⎣z11 z12 z13 ⎦ ⎣w3 ⎦ ⎣z3 ⎦
27
28
3 ANNs for Optimized Prediction
3.5.2 Forward Propagation
The input and output values of the samples shown in Figure 3.1 can be represented by the following
matrices:
⎡ 3 5⎤
X = ⎢ 5 1⎥
⎢
⎥
⎣10 2⎦
⎡75⎤
Y = ⎢82⎥ .
⎢ ⎥
⎣93⎦
Let us assume that the link weight values for the W (1) and W (2) matrices are assigned randomly
as follows:
W
(1) =
[
]
−1 3 1
2 −2 1
⎡−1⎤
W (2) = ⎢ 1⎥ .
⎢ ⎥
⎣ 2⎦
(3.3)
Using the X and W (1) matrices, we then have
] ⎡ 7 −1 −2⎤
⎡ 3 5⎤ [
−1 3 1
= ⎢−3 13 4⎥ .
Z (2) = X ⋅ W (1) = ⎢ 5 1⎥ ⋅
⎥
⎥
⎢
⎢
2 −2 −1
⎣−6 26 8⎦
⎣10 2⎦
Suppose a rectilinear activation function is employed in the neurons, which eliminates negative
values in Z (2) . We then compute a(2) as follows:
⎡7 0 0⎤
a(2) = f (Z (2) ) = ⎢0 13 4⎥ .
⎢
⎥
⎣0 26 8⎦
Using a(2) from above, Z (3) can be computed as follows:
⎡7 0 0⎤ ⎡−1⎤ ⎡ −7 ⎤
Z (3) = a(2) ⋅ W (2) = ⎢0 13 4⎥ ⋅ ⎢ 1⎥ = ⎢ 21⎥ .
⎢
⎥ ⎢ ⎥ ⎢
⎥
⎣0 26 8⎦ ⎣ 2⎦ ⎣ 42⎦
Then the ANN output for a rectilinear activation function will be (note that since the outputs are
grades, no negative values are allowed, and, hence, a rectilinear activation is acceptable):
⎡0⎤
ŷ = f (Z (3) ) = ⎢21⎥ .
⎢ ⎥
⎣42⎦
Error J for this iteration is half of the square of the difference between the real and computed
output values:
∑1
1
(y − ŷ )2 = ((75 − 0)2 + (82 − 21)2 + (93 − 42)2 ) = 5 974.
(3.4)
J=
2
2
3.5.3 Back Propagation
Let us now perform a back propagation starting from the output layer. Using the definition of 𝛿 (3)
in Eq. (2.13) from Section 2.6.3, we have
⎡0⎤ ⎡ 0 ⎤
⎡75⎤
𝛿 (3) = −(y − ŷ )⋅ ∗ f (Z (3) ) = − ⎢61⎥ ⋅ ∗ ⎢1⎥ = ⎢−61⎥ ,
⎥
⎢ ⎥ ⎢
⎢ ⎥
⎣1⎦ ⎣−51⎦
⎣51⎦
3.5 Case Study 1: Sleep-Study Example
where ⋅∗ represents element-wise multiplication. Based on the definition of 𝛿 (2) in Eq. (2.15) from
Section 2.6.3, we have
⎡ 0⎤ [
⎡1 0 0⎤
]
𝛿 (2) = 𝛿 (3) ⋅ W (2)† ⋅ ∗ f (Z (2) ) = ⎢−61⎥ ⋅ −1 −1 2 ⋅ ∗ ⎢0 1 1⎥
⎢
⎥
⎢
⎥
⎣−51⎦
⎣0 1 1⎦
0
0 ⎤
0 ⎤
⎡0
⎡1 0 0⎤ ⎡0 0
= ⎢61 −61 −122⎥ ⋅ ∗ ⎢0 1 1⎥ = ⎢0 −61 −122⎥ .
⎢
⎥
⎢
⎥ ⎢
⎥
⎣51 −51 −102⎦
⎣0 1 1⎦ ⎣0 −51 −102⎦
Using Eq. (2.14), we have
⎡7 0 0 ⎤ ⎡ 0 ⎤ ⎡ 0 ⎤
𝜕J
(2)†
(3)
⎢0 13 26⎥ ⋅ ⎢−61⎥ = ⎢−2119⎥ ,
=
a
⋅
𝛿
=
⎢
⎥ ⎢
⎥ ⎢
⎥
𝜕W (2)
⎣0 4 8 ⎦ ⎣−51⎦ ⎣ −652 ⎦
where † represents the matrix transpose operation. With Eq. (2.16), we have
[
] ⎡0
]
0
0 ⎤ [
3 5 10 ⎢
𝜕J
†
(2)
⎥ = 0 −815 −1630 .
=
X
⋅
𝛿
=
⋅
61
−61
−122
⎥
5 1 2 ⎢
0 −163 −326
𝜕W (1)
⎣51 −51 −102⎦
3.5.4 Updating Weights
Let the learning rate be 𝛼 = 0.001, where 𝛼 is a user-defined parameter to indicate how fast the
weights should be modified. Updating the weights at a rate of 𝛼 is referred to as learning. With
higher values of 𝛼, learning will be faster but possibly error prone due to taking larger steps to
modify. On the other hand, smaller 𝛼 values will be slower to converge to an acceptable solution
and will hence increase the cost of training. From Eq. (2.17), we have
[
]
[
]
−1 3 −1
0 −815 −1630
𝜕J
(1)
= W (1) − 𝛼
=
−
0.001
(3.5)
Wnew
2 −2 −1
0 −163 −326
𝜕W (1)
[
]
−1 3.8 2.6
=
2 −1.8 −0.7
(2)
= W (2) − 𝛼
Wnew
⎡ −1⎤
⎡ 0 ⎤ ⎡ −1 ⎤
⎢ ⎥
𝜕J
⎢−2119⎥ = ⎢ 3.1⎥ ,
=
−
0.001
1
⎢ ⎥
⎢
⎥ ⎢
⎥
𝜕W (2) ⎢ ⎥
⎣ −652 ⎦ ⎣ 2.7⎦
⎣ 2⎦
where the subscript new on the left-hand side of the equations denotes the updated values of the
weights.
3.5.5 Forward Propagation with New Weights
Let us now perform a new forward propagation using the new weights:
W (1) =
[
]
−1 3.8 2.6
2 −1.8 −0.7
⎡ −1 ⎤
W (2) = ⎢ 3.1⎥
⎢
⎥
⎣ 2.7⎦
29
30
3 ANNs for Optimized Prediction
] ⎡ 7 2.3 4.5 ⎤
⎡ 3 5⎤ [
−1 3.8 2.6
Z (2) = X ⋅ W (1) = ⎢ 5 1⎥ ⋅
= ⎢−3 17.3 12.5⎥
⎢
⎥
⎢
⎥
2 −1.8 −0.7
⎣10 2⎦
⎣−6 34.5 24.7⎦
⎡7 2.3 4.5 ⎤
a(2) = f (Z (2) ) = ⎢0 17.3 12.5⎥
⎢
⎥
⎣0 34.5 24.7⎦
⎡7 2.3 4.5 ⎤ ⎡ −1 ⎤ ⎡ 12 ⎤
Z (3) = a(2) ⋅ W (2) = ⎢0 17.3 12.5⎥ ⋅ ⎢ 3.1⎥ = ⎢ 87 ⎥
⎢
⎥ ⎢
⎥ ⎢ ⎥
⎣0 34.5 24.7⎦ ⎣ 2.7⎦ ⎣174⎦
⎡ 12 ⎤
ŷ = f (Z (3) ) = ⎢ 87 ⎥ .
⎢ ⎥
⎣174⎦
Error J at the end of second forward propagation will be
∑1
1
(y − ŷ )2 = ((75 − 12)2 + (82 − 87)2 + (93 − 174)2 ) = 5 273.
J=
2
2
Notice that the error is reduced to 5 273 from 5 974 in the previous iteration.
3.5.6 Back Propagation with New Weights
Continuing with back propagation using the new weights, we have
⎡1⎤ ⎡ 63 ⎤
⎡ 75 − 12 ⎤
𝛿 (3) = (y − ŷ ) ⋅ ∗ f (Z (3) ) = ⎢ 82 − 86 ⎥ ⋅ ∗ ⎢1⎥ = ⎢ −4 ⎥ ,
⎥
⎢ ⎥ ⎢
⎢
⎥
⎣1⎦ ⎣−81⎦
⎣93 − 174⎦
where ⋅∗ represents element-wise multiplication.
⎡1 1 1⎤
⎡ 63 ⎤ [
]
𝛿 (2) = 𝛿 (3) ⋅ W (2)† ⋅ ∗ f (Z (2) ) = ⎢ −4 ⎥ ⋅ −1 3.1 2.7 ⋅ ∗ ⎢0 1 1⎥
⎢
⎥
⎢
⎥
⎣0 1 1⎦
⎣−81⎦
⎡−6.3 196.6 166.9 ⎤
= ⎢ 0 −12.5 −10.6 ⎥
⎢
⎥
⎣ 51 −252.7 −214.6⎦
0 ⎤ ⎡ 63 ⎤ ⎡ 441 ⎤
⎡7 0
𝜕J
(2)†
(3)
⎢2.3 17.3 34.5⎥ ⋅ ⎢ −4 ⎥ = ⎢−2722⎥
=
a
⋅
𝛿
=
⎢
⎥ ⎢
⎥ ⎢
⎥
𝜕W (2)
⎣4.5 12.5 24.9⎦ ⎣−81⎦ ⎣−1785⎦
[
] ⎡−6.3 196.6 166.9 ⎤
3 5 10 ⎢
𝜕J
†
(2)
=
X
⋅
𝛿
=
⋅ 0 −12.5 −10.6 ⎥
⎥
5 1 2 ⎢
𝜕W (1)
⎣ 0 −252.7 −214.6⎦
[
]
−189 −1992 −1692
=
.
0
464
392
(3.6)
3.5 Case Study 1: Sleep-Study Example
The new weights after the second iteration become
𝜕J
(1)
Wnew
= W (1) − 𝛼
𝜕W (1)
[
]
[
]
−1 3.8 12.6
−189 −1992 −1692
=
− 0.001
2 −1.8 −0.7
0
464
392
[
]
−0.8 5.7 4.3
=
2.3 −1.4 −0.9
(2)
= W (2) − 𝛼
Wnew
(3.7)
⎡ −1⎤
⎡ 441 ⎤ ⎡−1.4⎤
⎢ ⎥
𝜕J
⎢−2722⎥ = ⎢ 5.8⎥ .
−
0.001
=
3.1
⎢ ⎥
⎢
⎥ ⎢
⎥
𝜕W (2) ⎢ ⎥
⎣−1785⎦ ⎣ 4.3⎦
⎣ 2.7⎦
3.5.7 Using Normalized Input and Output Values
To explore the impact of normalizing the input and output values, we go over the same forward
and back propagation operations from the previous section, but now using the normalized values.
Input values given in Figure 3.1 are normalized by dividing the first input x1 by 10 and the second
input x2 by 5; the outputs are all divided by 100. With this modification, the input and output values
become
⎡0.3 1 ⎤
X = ⎢0.5 0.2⎥
⎢
⎥
⎣ 1 0.4⎦
⎡0.75⎤
Y = ⎢0.82⎥ .
⎢
⎥
⎣ 093 ⎦
Randomly selected W (1) and W (2) matrices are
W (1) =
Z
(2)
[
]
−0.2 0.8 0.3
0.6 −0.4 −0.2
=X ⋅W
(1)
⎡−0.3⎤
W (2) = ⎢ 0.2⎥
⎢
⎥
⎣ 0.5⎦
]
⎡0.3 1 ⎤ [
−0.2 0.8 0.3
⎢
⎥
= 0.5 0.2 ⋅
⎢
⎥
0.6 −0.4 −0.2
⎣ 1 0.4⎦
⎡0.54 −0.16 −0.29⎤
= ⎢0.02 0.32 0.11⎥ .
⎢
⎥
⎣0.04 0.64 0.22⎦
Using a rectilinear activation function, we have
0 ⎤
⎡0.54 0
a(2) = f (Z (2) ) = ⎢0.02 0.32 0.11⎥
⎢
⎥
⎣0.04 0.64 0.22⎦
0 ⎤ ⎡−0.3⎤ ⎡−0.16⎤
⎡0.54 0
Z (3) = a(2) ⋅ W (2) = ⎢0.02 0.32 0.11⎥ ⋅ ⎢ 0.2⎥ = ⎢ 0.11⎥
⎢
⎥ ⎢
⎥ ⎢
⎥
⎣0.04 0.64 0.22⎦ ⎣ 0.5⎦ ⎣ 0.23⎦
⎡ 0 ⎤
ŷ = f (Z (3) ) = ⎢0.11⎥ .
⎢
⎥
⎣0.23⎦
(3.8)
(3.9)
31
32
3 ANNs for Optimized Prediction
Error J at the end of first forward propagation is
∑1
1
(y − ŷ )2 = ((0.75 − 0)2 + (0.82 − 0.11)2 + (0.93 − 0.23)2 ) = 0.778.
J=
2
2
(3.10)
Continuing with back propagation using the normalized inputs and outputs:
⎡0.75⎤
⎡0⎤ ⎡ 0 ⎤
𝛿 (3) = −(y − ŷ ) ⋅ ∗ f (Z (3) ) = − ⎢0.71⎥ ⋅ ∗ ⎢1⎥ = ⎢−0.71⎥ ,
⎢
⎥
⎢ ⎥ ⎢
⎥
⎣ 0.7 ⎦
⎣1⎦ ⎣ −0.7 ⎦
where ⋅∗ represents element-wise multiplication.
⎡1 0 0⎤
⎡ 0 ⎤ [
]
𝛿 (2) = 𝛿 (3) ⋅ W (2)† ⋅ ∗ f (Z (2) ) = ⎢−0.71⎥ ⋅ −0.3 0.2 0.5 ⋅ ∗ ⎢1 1 1⎥
⎢
⎥
⎢
⎥
⎣1 1 1⎦
⎣ −0.7 ⎦
0
0 ⎤
0
0 ⎤
⎡1 0 0⎤ ⎡ 0
⎡ 0
= ⎢0.21 −0.14 −0.36⎥ ⋅ ∗ ⎢1 1 1⎥ = ⎢0.21 −0.14 −0.36⎥
⎥
⎢
⎥ ⎢
⎢
⎥
⎣1 1 1⎦ ⎣0.21 −0.14 −0.35⎦
⎣0.21 −0.14 −0.35⎦
⎡0.54 0.02 0.04⎤ ⎡ 0 ⎤ ⎡−0.04⎤
𝜕J
(2)†
(3)
⎢ 0 0.32 0.64⎥ ⋅ ⎢−0.71⎥ = ⎢−0.68⎥
=
a
⋅
𝛿
=
⎢
⎥ ⎢
⎥ ⎢
⎥
𝜕W (2)
⎣ 0 0.11 0.22⎦ ⎣ −0.7 ⎦ ⎣−0.23⎦
[
] ⎡ 0
0
0 ⎤
0.3 0.5 1
𝜕J
†
(2)
⎢0.21 −0.14 −0.36⎥
=
X
⋅
𝛿
=
⋅
⎥
1 0.2 0.4 ⎢
𝜕W (1)
⎣0.21 −0.14 −0.35⎦
[
]
0.32 −0.21 −0.53
=
.
0.13 −0.08 −0.21
Let us now compute the weights for the normalized values using a learning rate of 𝛼 = 0.5 (value
is arbitrarily chosen):
[
]
[
]
−0.2 0.8 0.3
0.32 −0.21 −0.53
𝜕J
(1)
Wnew
= W (1) − 𝛼
=
−
0.5
0.6 −0.4 −0.2
0.13 −0.08 −0.21
𝜕W (1)
[
]
−0.36 0.91 0.57
=
0.54 −0.36 −0.1
(2)
= W (2) − 𝛼
Wnew
⎡−0.04⎤ ⎡−0.28⎤
⎡−0.3⎤
𝜕J
⎢ 0.2⎥ − 0.5 ⎢−0.68⎥ = ⎢ 0.54⎥ .
=
⎥
⎥ ⎢
⎥
⎢
𝜕W (2) ⎢
⎣−0.23⎦ ⎣ 0.62⎦
⎣ 0.5⎦
Let us continue another forward propagation using normalized inputs and outputs with the new
weight values:
⎡0.3 1 ⎤
X = ⎢0.5 0.2⎥
⎢
⎥
⎣ 1 0.4⎦
⎡0.75⎤
Y = ⎢0.82⎥ .
⎢
⎥
⎣ 093 ⎦
The newly updated W (1) and W (2) matrices are
W (1) =
[
]
−0.36 0.91 0.57
0.54 −0.36 −0.1
⎡−0.28⎤
W (2) = ⎢ 0.54⎥ .
⎢
⎥
⎣ 0.62⎦
(3.11)
3.5 Case Study 1: Sleep-Study Example
For the hidden layer, we have
Z
] ⎡ 0.43 −0.09 0.07⎤
⎡0.3 1 ⎤ [
−0.36 0.91 0.57
⎥
= ⎢−0.07 0.38 0.27⎥ .
0.5 0.2 ⋅
⎢
⎥
⎢
⎥
0.54 −0.36 −0.1
⎣ 1 0.4⎦
⎣−0.14 0.77 0.53⎦
(2) = X ⋅ W (1) = ⎢
Using a rectilinear activation function, we have
⎡0.43 0 0.07⎤
a(2) = f (Z (2) ) = ⎢ 0 0.38 0.27⎥
⎢
⎥
⎣ 0 0.77 0.53⎦
⎡0.43 0 0.07⎤ ⎡−0.28⎤ ⎡−0.08⎤
Z (3) = a(2) ⋅ W (2) = ⎢ 0 0.38 0.27⎥ ⋅ ⎢ 0.54⎥ = ⎢ 0.37⎥
⎥
⎥ ⎢
⎢
⎥ ⎢
⎣ 0 0.77 0.53⎦ ⎣ 0.62⎦ ⎣ 0.74⎦
⎡ 0 ⎤
ŷ = f (Z (3) ) = ⎢0.37⎥ .
⎢
⎥
⎣0.74⎦
After the second forward propagation with normalized inputs and output, we have error J as
follows:
∑1
1
J=
(3.12)
(y − ŷ )2 = ((0.75 − 0)2 + (0.82 − 0.37)2 + (0.93 − 0.74)2 ) = 0.4.
2
2
The error value of 0.778 obtained at the previous iteration is now reduced to 0.4. Continuing with
back propagation:
𝛿
(3)
⎡0⎤ ⎡ 0 ⎤
⎡ 0.75 − 0 ⎤
(3)
⎢
⎥
̂
= −(y − y) ⋅ ∗ f (Z ) = − 0.82 − 0.37 ⋅ ∗ ⎢1⎥ = ⎢0.45⎥ ,
⎥
⎢ ⎥ ⎢
⎢
⎥
⎣1⎦ ⎣0.19⎦
⎣0.93 − 0.74⎦
where ⋅∗ represents element-wise multiplication.
⎡ 0 ⎤[
⎡1 0 1⎤
]
𝛿 (2) = 𝛿 (3) ⋅ W (2)† ⋅ ∗ f (Z (2) ) = ⎢0.45⎥ −0.28 0.54 0.62 ⋅ ∗ ⎢0 1 1⎥
⎢
⎥
⎢
⎥
⎣0.19⎦
⎣0 1 1⎦
0 ⎤
⎡0 0
= ⎢0 0.24 0.28⎥
⎢
⎥
⎣0 0.1 0.12⎦
0 ⎤ ⎡ 0 ⎤ ⎡ 0 ⎤
⎡0.43 0
𝜕J
(2)†
(3)
⎢ 0 0.38 0.77⎥ ⋅ ⎢0.45⎥ = ⎢0.32⎥
=
a
⋅
𝛿
=
⎥
⎥ ⎢
⎢
⎥ ⎢
𝜕W (2)
⎣0.07 0.27 0.53⎦ ⎣0.19⎦ ⎣0.22⎦
]
[
] ⎡0 0
0 ⎤ [
0.3 0.5 1
𝜕J
†
(2)
⎢0 0.24 0.28⎥ = 0 0.22 0.26 .
=
X
⋅
𝛿
=
⋅
⎥
0 0.09 0.1
1 0.2 0.4 ⎢
𝜕W (1)
⎣0 0.1 0.12⎦
Let us now compute the weights for the normalized values using a learning rate arbitrarily set as
𝛼 = 0.5:
[
]
[
]
−0.36 0.91 0.57
0 0.22 0.26
𝜕J
(1)
Wnew
= W (1) − 𝛼
=
−
0.5
0.54 −0.36 −0.1
0 0.09 0.1
𝜕W (1)
(3.13)
[
]
−0.36
0.8 0.44
=
0.54 −0.41 −0.15
33
34
3 ANNs for Optimized Prediction
(2)
Wnew
= W (2) − 𝛼
⎡ 0 ⎤ ⎡−0.28⎤
⎡−0.28⎤
𝜕J
⎢ 0.54⎥ − 0.5 ⎢0.32⎥ = ⎢ 0.38⎥ .
=
⎥
⎢
⎥ ⎢
⎥
𝜕W (2) ⎢
⎣0.22⎦ ⎣ 0.51⎦
⎣ 0.62⎦
3.5.8 Reducing Errors During Training
It is expected that during training, after each epoch, the difference between the real and computed
output values becomes smaller, implying that the weights are adjusted properly and that, when
real input values are applied, the trained ANN will generate outputs that are close to the real values. Below, we repeat the error values derived above to explore the convergence of the computed
output values toward the real ones both for non-normalized and normalized input and output values. For the non-normalized case, after the first forward propagation, from Eq. (3.4), we have the
error as
∑1
1
J=
(y − ŷ )2 = ((75 − 0)2 + (82 − 21)2 + (93 − 42)2 ) = 5 974.
2
2
For the second iteration from Eq. (3.6), error is
∑1
1
(y − ŷ )2 = ((75 − 12)2 + (82 − 87)2 + (93 − 174)2 ) = 5 273.
J=
2
2
For the normalized case, the first forward propagation error from Eq. (3.10) is
∑1
1
(y − ŷ )2 = ((0.75 − 0)2 + (0.82 − 0.11)2 + (0.93 − 0.23)2 ) = 0.778.
J=
2
2
For the second normalized forward propagation, from Eq. (3.12), we have
∑1
1
(y − ŷ )2 = ((0.75 − 0)2 + (0.82 − 0.37)2 + (0.93 − 0.74)2 ) = 0.4.
J=
2
2
We observe that the error values get smaller regardless of the normalization of the sample data.
It is not, however, conclusive from this limited runtime that normalization yields better results.
We expect that as we perform more epochs, the benefits of normalization will quickly become
apparent.
3.5.9 Implementation of Sleep-Study ANN in Python
A sample implementation of an ANN in Python for the sleep-study example is given below. In line
31, the user is asked whether a new ANN will be trained or an existing already trained ANN will be
used for prediction by entering 1 or 2, respectively. If the user chooses to train a new ANN by entering 1, setting up the ANN parameters for training starts in line 42. In line 50, a multi-dimensional
array called df is created by reading the input file defined in line 44. In lines 52 and 54, the ANN
inputs and outputs are saved into arrays called X and Y, respectively. In line 57, a model for the
ANN is created using a function named Sequential as follows:
model.add(keras.layers.Dense(3,activation=’relu’,input_shape=(,2)))
where the new ANN will be fully-connected (i.e. option Dense) with three layers using rectilinear
activation and with two inputs. In line 66, an output layer is added to the model object:
model.add(keras.layers.Dense(3,activation=’linear’))
3.5 Case Study 1: Sleep-Study Example
In line 70, the error function will be minimized using the gradient descent algorithm (i.e. option
adam) using the squares of the difference between the real and computed outputs (i.e. option
mean_squared_error) as follows:
model.compile(optimizer=’adam’,loss=’mean_squared_error’)
In line 73, training is executed by
model.fit(X, Y,epochs=2000)
which indicates that input and output data are in X and Y arrays, respectively, and training will be
performed for 2000 epochs.
If an existing ANN is to be used for prediction (i.e. user enters 2 at line 36), its name is saved in
the load_file variable and is prepared to run in line 87 by
model = keras.models.load_model(load_file)
The final values of weights either after training is complete or for the saved ANN are printed
in lines 95 and 96. Between lines 101 and 134, a prediction is obtained using the selected ANN
for new user inputs. In lines 103 and 104, the user is asked to enter new values for the number of hours studied and slept, respectively, which are saved in an array called user_input. A
prediction is generated by the ANN in line 109 and saved into a variable called prediction in
line 109:
prediction = model.predict(user_input)
In lines 138 to 152, the script asks if the ANN model will be saved for later use and, if so, saves
its name in a variable called bf save_name.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# The City College of New York, City University of New York
# Written by Ricardo Valdez
# August, 2020
# SLEEP STUDY EXAMPLE
# An ANN to predict the exam grade for a student based on the number of hours
# slept and studied for the exam
# This example problem is from: https://www.youtube.com/watch?v=bxe2t-v8xrs
import numpy as np
import pandas as pd
import time
from tensorflow import keras
def print_weights(weights):
# weights = model.get_weights();
print('\n******* WEIGHTS OF ANN *******\n')
for i in range(int(len(weights)/2)):
print('Weights W%d:\n' %(i), weights[i*2])
print('Bias b%d:\n' %(i), weights[(i*2)+1])
#END print_weights()
#% ANN TRAINING
print('\n')
print('*********************************************************************')
print('** WELCOME TO GRADE PREDICTIONS USING ARTIFICIAL NEURAL NETWORKS **')
print('*********************************************************************')
# prompt user to train or load an ANN model
35
36
3 ANNs for Optimized Prediction
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
option_list = ['1','2']
option = ''
while option not in option_list:
print('\nOPTIONS:')
print('1 - Train a new ANN model')
print('2 - Load an existing model')
option = input('\nSelect an option by entering a number: \n')
if option not in option_list:
message = 'Invalid input: Input must be one of the following - '
print(message, option_list)
time.sleep(2)
if option == '1':
## OPTION 1: TRAIN A NEW ANN MODEL
train_data_file = 'study_data.csv'
print('\n********* NOW TRAINING ANN USING', train_data_file,'*********')
time.sleep(3)
## load the training data
df = pd.read_csv(train_data_file)
## define input matrix X (get rid of column called exam_grade)
X = np.array(df.drop(['exam_grade'], axis=1))
## define expected output matrix Y
Y = np.array(df['exam_grade'])
## create a model for the ANN
model = keras.Sequential()
## add a hidden layer that accepts 2 input features (hours studied/slept)
## the hidden layer has 3 neurons.
## Dense means every neuron in the layer connects to every neuron in the
## previous layer.
model.add(keras.layers.Dense(3, activation='relu', input_shape=(2,)))
# ## Add another hidden layer with 4 neurons to the ANN
# model.add(keras.layers.Dense(4, activation='relu')
## add an output layer with a single output (exam grade)
model.add(keras.layers.Dense(1, activation='linear'))
## set the optimization algorithm used for minimizing loss function
## use gradient descent (adam) to minimize error (loss)
model.compile(optimizer='adam', loss='mean_squared_error')
## train the ANN model using 2000 iterations
model.fit(X, Y, epochs=2000)
print('\n\n********** ANN training complete **********\n\n')
elif option == '2':
## OPTION 2: LOAD ANN MODEL FROM FILE
message = 'Enter the file name of the ANN Model you want to load: \n'
load_file = input(message)
#load_file = input('It must be a .h5 file')
## if file name does not end with '.h5', add '.h5' to the file name
if load_file[-3:] != '.h5':
load_file += '.h5'
## load the ANN model from load_file
model = keras.models.load_model(load_file)
print('\n\n****** SUCCESSFULLY LOADED ANN MODEL FROM', load_file,'******')
else:
print('ERROR: INVALID OPTION SELECTED')
## raise an exception to terminate the program
raise ValueError()
weights = model.get_weights();
print_weights(weights)
#% GRADE PREDICTION USING ANN
input('\n\n********** Press ENTER to start using the ANN **********\n\n')
finished = False
while not finished:
3.5 Case Study 1: Sleep-Study Example
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
## prompt user for inputs
studied = float(input('\n\nEnter number of hours studied: \n'))
slept = float(input('Enter number of hours slept: \n'))
# ## get ANN prediction (only element [0,0])
# prediction = model.predict([[studied,slept]])[0,0]
user_input = np.array([[studied,slept]])
prediction = model.predict(user_input)
## restrict prediction between 0 and 100
if prediction > 100:
prediction = 100
elif prediction < 0:
prediction = 0
else:
pass
## display prediction
print('\n*******************************')
print('ANN Predicted Grade: ', int (prediction))
print('*******************************')
## ask user if they would like to continue
choice = ''
while choice not in ['y','n']:
choice = input('\n\nWould you like to continue? (y/n): \n')
if choice == 'y':
pass
elif choice == 'n':
finished = True
else:
print("Invalid input: Input must be 'y' or 'n'")
#end while
#end while
# ask user if they would like to save the ANN model
choice = ''
while choice not in ['y','n']:
choice = input('\n\nWould you like to save the ANN model? (y/n): \n')
if choice == 'y':
save_name = input('\n\nEnter a name for the save file: \n')
## if file name does not end with '.h5', add '.h5' to the file name
if save_name[-3:] != '.h5':
save_name += '.h5'
model.save(save_name)
print('\n\n')
print('***** ANN MODEL SUCCESSFULLY SAVED AS '+save_name+' *****')
elif choice == 'n':
pass
else:
print("Invalid input: Input must be 'y' or 'n'")
#end while
Data used in this script is given in Figure 3.3(a). In realistic applications, thousands or even millions of samples are employed. In the sample run shown in Figure 3.3(b), the user enters 7 and
3 as the number of hours studied and slept, respectively, for which the ANN predicts a grade
of 98.
3.5.10 Implementation of Sleep-Study ANN in Matlab
A Matlab implementation of the sleep-study ANN is presented below. The structure of the script,
variables and functions are kept similar to the Python script presented in Section 3.5.9 so that the
reader can easily compare the different implementations.
In lines 15–26, a rudimentary user interaction module is written asking the user to either train
a new ANN or use an already-trained one. If the user enters 1, in line 35 a multidimensional array
37
38
3 ANNs for Optimized Prediction
(a)
(b)
Figure 3.3 (a) Sample data used for sleep-study example, and (b) screenshot generated by Python script,
where the user enters 7 and 3 as the number of hours studied and slept, respectively, for which the ANN
predicts grade of 98.
called df is created by reading an input file. Sample inputs of X and outputs of Y are set in lines
37 and 39, respectively. Matlab uses some percentage of the sample data to train an ANN and the
remaining samples to test the accuracy of the training. These percentages are set in lines 60, 63
and 70 for the percentages of data to be used for training, validation and testing, respectively. The
number of inputs is set in line 43 and the number of hidden layers in line 45. An ANN object called
net is created in line 47 by
net = fitnet(hiddenLayerSize);
In line 73, the ANN object net is trained using inputs and outputs:
[net, tr] = train(net, X, Y);
where tr is a structure that keeps a training record of details such as performance, best performing epoch and intermediate variable values. If the user entered 2, the name of a trained
ANN is requested in line 78 and is loaded into the net object in line 84. At this point net
contains an ANN, either a newly or previously trained one. Between lines 95 and 120, there is
a loop that keeps asking the user to enter two inputs (study hours kept in variable studied
in line 97 and sleep hours in slept line 98) and generates an output saved in prediction in
line 100 as
prediction = net(studied, slept);
Every time the user enters y in line 177, the loop repeats asking for study hours in line 97
and sleep in line 98, generating a new prediction in line 100 as above. The loop terminates
when the user enters n in line 177. If the user enters y in line 124, the trained ANN is saved in
lines 125 to 127.
Figure 3.4 shows a screenshot generated by the Matlab script where the user enters 7 and 3 as
the number of hours studied and slept, respectively, for which the ANN predicts a grade of 82. This
result is slightly different than the one from the Python script in Figure 3.3, which predicts the
grade to be 98 for 7 hours of study and 3 hours of sleep. This is to be expected since the libraries
used in Python and Matlab have different internal implementations for ANN training.
3.5 Case Study 1: Sleep-Study Example
Figure 3.4 Sample screenshot generated by the Matlab script for the sleep-study example, where the
user enters 7 and 3 as the number of hours studied and slept, respectively, for which the ANN predicts a
grade of 82.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
% The City College of New York, City University of New York
% Written by Olga Chsherbakova
% Date: November, 2023
% Prefiction of exam grade using an ANN based on number of sleep and study
clc;
clear;
% ANN TRAINING
disp('*******************************************************************')
disp('**
WELCOME TO GRADE PREDICTIONS USING ANNS
**')
disp('*******************************************************************')
% prompt user to train or load an ANN model
option = '';
while ∼any(strcmp(option, {'1', '2'}))
disp('OPTIONS:');
disp('1 - Train a new ANN model');
disp('2 - Load an existing model');
option = input('\nSelect an option by entering a number: ', 's');
if ∼any(strcmp(option, {'1', '2'}))
disp('Invalid input: Input must be {''1'', ''2''}');
pause(2);
end
end
if strcmp(option, '1')
% OPTION 1: TRAIN A NEW ANN MODEL
train_data_file = 'study_data.csv';
disp(['***** NOW TRAINING ANN USING ', train_data_file, ' *********']);
pause(3);
% load the training data
df = readtable(train_data_file);
% define input matrix X (get rid of column called exam_grade)
X = df{:, {'hours_studied', 'hours_slept'}}';
% define expected output matrix Y
Y = df{:, 'exam_grade'}';
% create a model for the ANN
% Two inputs (hours to study and hours to sleep)
inputSize = 2;
% Add two hidden layers
hiddenLayerSize = 2;
% fit function
net = fitnet(hiddenLayerSize);
% The reason for this data split is to avoid overfitting, which is when
% a model becomes too specific to the training data and doesn't
39
40
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
3 ANNs for Optimized Prediction
% generalize well to new data. By having a separate validation set,
% you can monitor the model's performance during training and
% make adjustments if necessary. The test set is used to evaluate
% the model's performance in a final assessment.
% sets the ratio of the dataset used for training.
% In this case, 70% of the data is used for training the ANN.
% This is the data that the ANN learns from to update its weights
% and minimize the training error.
net.divideParam.trainRatio = 70/100;
% sets the ratio of the dataset used for validation. In this case, 15%
% of the data is used for validation. The validation set is used to
% monitor the ANN's performance during training.
% It helps to detect overfitting and allows fine-tune hyperparameters.
net.divideParam.valRatio = 15/100;
% sets the ratio of the dataset used for testing. In this case, 15%
% of the data is reserved for testing the trained ANN. The test set is
% used to evaluate the final performance of the trained ANN on data
% it has never seen before.
net.divideParam.testRatio = 15/100;
% train the ANN model
[net, tr] = train(net, X, Y);
disp('********** ANN training complete **********');
elseif strcmp(option, '2')
% OPTION 2: LOAD ANN MODEL FROM FILE
load_file = input('Enter the file name of ANN model to load: ', 's');
if ∼endsWith(load_file, '.mat')
load_file = [load_file '.mat'];
%load_file = input('It must be a .mat file')
% if file name does not end with '.mat', add '.mat'
end
load(load_file, 'net');
disp(['**** SUCCESSFULLY LOADED ANN MODEL FROM ', load_file, ' ****']);
end
% Display the nets of the models
% disp(net)
% GRADE PREDICTION USING ANN
input('********** Press ENTER to start using the ANN **********', 's');
finished = false;
while ∼finished
% prompt user for inputs
studied = input('Enter number of hours studied: ');
slept = input('Enter number of hours slept: ');
prediction = net([studied; slept]);
% restrict prediction between 0 and 100
prediction = min(max(prediction, 0), 100);
% display prediction
fprintf('ANN Predicted Grade: %d\n', round(prediction));
% ask user if they would like to continue
choice = '';
while ∼any(strcmp(choice, {'y', 'n'}))
choice = input('Would you like to continue? (y/n): ', 's');
if strcmp(choice, 'y')
% Do nothing and continue the loop
elseif strcmp(choice, 'n')
finished = true;
else
disp("Invalid input: Input must be 'y' or 'n'");
end
end
end
% ask user if they would like to save the ANN model
choice = input('Would you like to save the ANN model? (y/n): ', 's');
3.6 Case Study 2: Prediction of Bike Rentals
124
125
126
127
128
129
130
if strcmp(choice, 'y')
save_file = input('Enter the file name to save the ANN Model: ', 's');
save([save_file '.mat'], 'net');
disp(['**** ANN MODEL SUCCESSFULLY SAVED TO ', save_file, '.mat ***']);
end
disp('Thank you for using the ANN for grade prediction. Goodbye!');
3.6 Case Study 2: Prediction of Bike Rentals
In this section, we present an ANN designed to predict the number of new bike rentals in a given
hour in London, UK. As shown in Figure 3.5, there are five inputs defined for the ANN, namely
the time of the day, temperature, wind speed, a weather code that describes the current climate
conditions and a Boolean value indicating whether the day is a weekend or not. The ANN output
is the number of bike rentals for a given hour. These input values are measurements observed
in the field, which are used as inputs without processing (e.g. finding minimum, maximum,
average, and so on). The ANN architecture is fully connected and consists of an input layer with
five inputs, two hidden layers with five and six neurons, and one output layer for the predicted
number of bike rentals in the next hour. Figure 3.6 illustrates the ANN layers for the bike rental
example, where most of the links between layers are omitted in the figure for simplicity. The
training process is similar to the one described above for the sleep-study example in Section 3.5,
with each epoch consisting of a forward propagation to make a prediction followed by adjustment
of the ANN weights and bias via back propagation. The number of epochs in this simple is kept as
2 000. Real-life ANN training typically uses up to millions of data samples and tens of thousands
of epochs.
3.6.1 Python Script for Bike Rentals Using an ANN
A sample Python script implementing the ANN in Figure 3.6 is presented below. In lines 32–125,
user selection of either to train a new ANN or load an existing one is evaluated. If user enters 1 as
a selection, in line 51, data from bike rentals in London [38], of which a small sample is displayed
in Figure 3.7, is defined as the training data saved as a multidimensional array of df.
In line 77, a function which converts a time-stamp to the hour of the day is invoked. All
time-stamps from sample data are converted to hours of the day in line 80. An ANN model is
true
Figure 3.5 ANN training data with five inputs and one output to be used for bike rentals example
(Data taken from [38]).
41
42
3 ANNs for Optimized Prediction
Figure 3.6 A fully connected ANN for bike rental example using two hidden layers (for simplicity, many
links are omitted in the figure).
Figure 3.7
Field data used for training the ANN shown in Figure 3.6.
created in line 89, and a hidden layer with five input neurons as Dense (i.e. fully connected) type
with rectilinear activation is created in line 95: Sample inputs and outputs are stored in X and Y
arrays in lines 84 and 86, respectively. In line 89 an object called model is created. Two hidden
layers with five and six neurons using rectilinear activation (option relu) are added to the ANN
in lines 95 and 97, respectively. An output layer, consisting of one neuron with a linear activation
function (i.e. without filtering the values during activation) is defined in line 99. The optimization
3.6 Case Study 2: Prediction of Bike Rentals
function is set as mean squared error using a gradient descent method in line 103. This ANN is
trained at line 105 by
model.fit(X, Y, epochs=2000)
When the user selects option 2 in lines 108 to 121, a saved ANN is loaded into the ANN
object called model. The weights of the selected ANN (either newly trained or saved one) are
printed on the screen using the scripts given in lines 127 and 128 by calling a function named
print_weights in lines 15–21, which prints the weights of the ANN.
Prediction of the number of bikes for a given day using the ANN is presented in the loop between
lines 133 and 171. The user is asked to enter the input values for a specific day, which are the inputs
to the ANN. In line 148, a prediction is generated for the user input data by
prediction = model.predict(user_input)
where the object called model contains the ANN, and user_input is the input variable populated by the values entered by user. In lines 175 and 187, the user is given an option to save the
current ANN.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# The City College of New York, City University of New York
# Written by Ricardo Valdez and Jian Wen Choong
# August, 2020
# Data for this example was taken from:
# https://www.kaggle.com/hmavrodiev/london-bike-sharing-dataset
import numpy as np
import pandas as pd
import time
from datetime import datetime
from tensorflow import keras
# import print_weights as pw
# call: pw.print_weights()
def print_weights(weights):
# weights = model.get_weights();
print('\n******* WEIGHTS OF ANN *******\n')
for i in range(int(len(weights)/2)):
print('Weights W%d:\n' %(i), weights[i*2])
print('Bias b%d:\n' %(i), weights[(i*2)+1])
#END print_weights()
#% ANN TRAINING
print('\n')
print('********************************************************************')
print('**** WELCOME TO BIKE SHARING USING ARTIFICIAL NEURAL NETWORKS ****')
print('********************************************************************')
# prompt user to train or load an ANN model
option_list = ['1','2']
option = ''
while option not in option_list:
print('\nOPTIONS:')
print('1 - Train a new ANN model')
print('2 - Load an existing model')
option = input('\nSelect an option by entering a number: \n')
if option not in option_list:
message = 'Invalid input: Input must be one of the following - '
print(message, option_list)
time.sleep(2)
if option == '1':
## OPTION 1: TRAIN A NEW ANN MODEL
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
3 ANNs for Optimized Prediction
train_data_file = 'london_bike_sharing_data.csv'
print('\n********* NOW TRAINING ANN USING', train_data_file,'*********')
time.sleep(3)
## load the training data
df = pd.read_csv(train_data_file)
## the training data contains 7 columns:
##
##
timestamp - the date and time the sample was recorded
##
new_bikes_shared - number of new bikes shared over the last hour
##
is_weekend - boolean that is 1 (true) if the day is a weekend
##
temp_c - the temperature in Celcius
##
wind_speed - wind speed in km/h
##
weather_code - category of weather: 1 = clear
##
2 = scattered clouds
##
3 = broken clouds
##
4 = cloudy
##
7 = rain
##
10 = thunderstorm
##
26 = snow
##
94 = freezing fog
## the timestamp column of df are stored as strings. We want to
## convert each timestamp string into a datetime objects using the
## function datetime.strptime(). The first input of datetime.strptime()
## is the string you want to convert, and the second input is the
## format of the string, where
## %m = month, %d = day, %Y = year, %H = hour, %M = minute.
##
## create lambda function to perform conversion and return the hour.
get_hour = lambda timestamp: datetime.strptime(timestamp,
'%m/%d/%Y %H:%M').hour
## apply the lambda function to every timestamp in column df['timestamp']
df['time_hour'] = df['timestamp'].apply(get_hour)
## define input matrix X (get rid of columns called timestamp and
## new_bikes_shared)
X = np.array(df.drop(['timestamp','new_bikes_shared'], axis=1))
## define expected output matrix Y
Y = np.array(df['new_bikes_shared'])
## create a model for the ANN
model = keras.Sequential()
## add a hidden layer that accepts 5 input features (time_hour, temp_c
## wind_speed, weather_code, is_weekend)
## the hidden layer has 5 neurons.
## Dense means every neuron in the layer connects to every neuron in the
## previous layer.
model.add(keras.layers.Dense(5, activation='relu', input_shape=(5,)))
## add another hidden layer with 6 neurons to the ANN
model.add(keras.layers.Dense(6, activation='relu'))
## add an output layer with a single output (new_bikes_shared)
model.add(keras.layers.Dense(1, activation='linear'))
## set the optimization algorithm used for minimizing loss function
## use gradient descent (adam) to minimize error (loss)
model.compile(optimizer='adam', loss='mean_squared_error')
## train the ANN model using 2000 iterations
model.fit(X, Y, epochs=2000)
print('\n\n********** ANN training complete **********\n\n')
elif option == '2':
## OPTION 2: LOAD ANN MODEL FROM FILE
message = 'Enter the file name of the ANN Model you want to load: \n'
load_file = input(message)
#load_file = input('It must be a .h5 file')
## if file name does not end with '.h5', add '.h5' to the file name
if load_file[-3:] != '.h5':
load_file += '.h5'
3.6 Case Study 2: Prediction of Bike Rentals
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
## load the ANN model from load_file
model = keras.models.load_model(load_file)
print('\n\n****** SUCCESSFULLY LOADED ANN MODEL FROM', load_file,'******')
else:
print('ERROR: INVALID OPTION SELECTED')
## raise an exception to terminate the program
raise ValueError()
weights = model.get_weights();
print_weights(weights)
#% BIKE SHARING PREDICTION USING ANN
input('\n\n********** Press ENTER to start using the ANN **********\n\n')
finished = False
while not finished:
## prompt user for inputs
temp_c = float(input('\n\nEnter temperature in Celcius: \n'))
hour = float(input('Enter hour of the day (military): (0-23) \n'))
is_weekend = input('Is it the weekend? (y/n): \n')
if is_weekend == 'y':
is_weekend = 1
else:
is_weekend = 0
wind_speed = float(input('Enter wind speed: (km/h) \n'))
weather_code = int(input('Enter weather code: (1 = clear, 2 = few clouds, '
+ '3 = broken clouds, 4 = cloudy, 7 = rain, '
+ '10 = thunderstorm, 26 = snow, 94 = freezing fog) \n'))
user_input=np.array([[is_weekend, temp_c, wind_speed, weather_code,hour]])
prediction = model.predict(user_input)
## restrict prediction to non-negative values
if prediction < 0:
prediction = 0
else:
pass
## display prediction
print('\n*****************************************')
print('ANN Predicted number of shared bikes: ', int (prediction))
print('*****************************************')
## ask user if they would like to continue
choice = ''
while choice not in ['y','n']:
choice = input('\n\nWould you like to continue? (y/n): \n')
if choice == 'y':
pass
elif choice == 'n':
finished = True
else:
print("Invalid input: Input must be 'y' or 'n'")
#END WHILE
#END WHILE
## ask user if they would like to save the ANN model
choice = ''
while choice not in ['y','n']:
choice = input('\n\nWould you like to save the ANN model? (y/n): \n')
if choice == 'y':
save_name = input('\n\nEnter a name for the save file: \n')
## if file name does not end with '.h5', add '.h5' to the file name
if save_name[-3:] != '.h5':
save_name += '.h5'
model.save(save_name)
print('\n\n')
print('***** ANN MODEL SUCCESSFULLY SAVED AS '+save_name+' *****')
elif choice == 'n':
pass
else:
print("Invalid input: Input must be 'y' or 'n'")
#END WHILE
45
46
3 ANNs for Optimized Prediction
Figure 3.8 Python script for bike rentals example predicts 822 bike rentals for a weekend with
temperature of 15 degrees C, cloudy skies, wind speed of 5 mph, at 10:00 am.
A screenshot from a sample run for this ANN is shown in Figure 3.8. In this run, the trained
ANN generates a prediction of 822 bike rentals for a weekend day with temperature of 15 degrees
C, cloudy skies, wind speed of 5 mph, at 10:00 am.
3.6.2 Matlab Script for Bike Rentals Using an ANN
A sample Matlab script implementing the bike rental ANN shown in Figure 3.6 is presented below.
The Matlab script has a similar structure to the Python implementation in Section 3.6.1 for ease of
comparison by the reader.
In line 15, the user is asked either to train a new ANN or load a saved one. If the user selects to
train a new one by entering 1, a multidimensional matrix called df is created in line 21 using the
sample data from a file. Input and output values are saved as X and Y in lines 29 and 31, respectively.
In line 34, an ANN object called net is created, which is trained line 36 by
net = train(net, X, Y);
If the user enters 2 in line 15, a previously trained ANN is loaded into the net object in line 42.
There is a loop between lines 56 and 66, where the trained ANN (either the new one or the saved
one) in net is executed based on user inputs, until user exits it by typing n in line 65. In lines 51–58,
user inputs for the ANN are requested and recorded. Then in line 60, a prediction is generated by
the ANN as follows:
prediction = net(is_weekend, time_c, wind_speed, weather_code,
hour);
where the ANN output is saved in prediction. In lines 69–74, the user is given the choice to
save the current ANN before exiting the script.
Figure 3.9 shows a sample run of the ANN implemented by the Matlab script, where for a weekend day with a temperature of 15 degrees C, under cloudy skies, for a wind speed of 5 mph and at
3.6 Case Study 2: Prediction of Bike Rentals
Figure 3.9 Matlab script for bike rentals given above predicts 928 rentals for a weekend with temperature
of 15 degrees C, cloudy skies, wind speed of 5 mph, at 10:00 am.
10:00 am, the prediction is 928 bike rentals. This result is not exactly the same as the prediction from
the Python script in Section 3.6.1 since implementations of library functions are different between
the two implementations. However, both results are consistent with the sample data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
% The City College of New York, City University of New York
% Written by Olga Chsherbakova
% November, 2023
% Prediction of Bike Sharing Using ANNs
clc;
clear;
% ANN TRAINING
disp('****************************************************************');
disp('** WELCOME TO BIKE SHARING USING ARTIFICIAL NEURAL NETWORKS **');
disp('****************************************************************');
option = input('Select an option (1- Train, 2- Load): ', 's');
if option == '1'
% prompt user to train or load an ANN model
train_data_file = 'london_bike_sharing_data.csv';
disp(['Training ANN using ', train_data_file]);
df = readtable(train_data_file);
% Convert timestamps
time_hour = hour(datetime(df.timestamp, ...
'InputFormat', 'MM/dd/yyyy HH:mm'));
df.time_hour = time_hour;
% define input matrix X and output matrix Y
X = [df.is_weekend, df.temp_c, df.wind_speed, df.weather_code, ...
df.time_hour];
Y = df.new_bikes_shared;
% Creation and training of a neural network
net = fitnet([5 6]);
% define expected output matrix Y
net = train(net, X', Y');
disp('Training complete.');
elseif option == '2'
% load the training data
load_file = input('Enter the model file name: ', 's');
load(load_file, 'net');
disp(['Model loaded from ', load_file]);
end
disp('Press ENTER to start using the ANN');
47
48
3 ANNs for Optimized Prediction
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
pause;
% prompt user to train or load an ANN model
finished = false;
while ∼finished
temp_c = input('Enter temperature in Celsius: ');
hour = input('Enter hour of the day (military): (0-23) ');
is_weekend = input('Is it the weekend? (y/n): ', 's');
is_weekend = double(is_weekend == 'y');
wind_speed = input('Enter wind speed (km/h): ');
disp('Weather code: 1 = clear, 2 = fw cldy, 3 = brk cldy, 4 = cloudy');
disp('7 = rain, 10 = thunderstorm, 26 = snow, 94 = freezing fog');
weather_code = input('Enter weather code: ');
prediction = net([is_weekend; temp_c; wind_speed; weather_code; hour]);
prediction = max(prediction, 0); % Limiting prediction to non-negative
disp(['Predicted number of shared bikes:',num2str(round(prediction))]);
choice = input('Would you like to continue? (y/n): ', 's');
finished = (choice == 'n');
end
% Save the model
choice = input('Would you like to save the ANN model? (y/n): ', 's');
if choice == 'y'
save_name = input('Enter a name for the save file: ', 's');
save([save_name '.mat'], 'net');
disp(['Model saved as ', save_name]);
end
3.7 Concluding Remarks
In this chapter, we present two simplified case studies for optimized prediction of events using
ANNs. The first is an ANN for predicting an exam grade based on the number of hours that a
student studied and slept for this exam. For this simplistic example, we present forward and back
propagation operations in detail, showing the values of parameters at every step of training with the
formulas derived in Chapter 2. The second example involves constructing and training an ANN to
predict the number of bike rentals based on sample data such as the day of the week, the time of the
day, air temperature and wind speed. This example is inspired by bike rental companies operating
in London, UK.
We present sample Python and Matlab scripts for implementation of these ANNs. Although we
use the most simple ANN architectures (minimal number of hidden layers with only a few neurons)
and very few data points, the ANNs generate predictions compatible with the training data, which
is a clear indication of the power of the ANN-based approach to optimized prediction.
3.8 Exercises for Chapter 3
1) Modify the Python script for the sleep-study ANN in Section 3.5.9 by adding more hidden
layers. What is the impact, if any?
2) Modify the Python script for the sleep-study ANN in Section 3.5.9 by increasing the number
of neurons. What is the impact, if any?
3) Modify the Python script for the sleep-study ANN in Section 3.5.9 by increasing the number
of hidden layers and neurons. What is the impact, if any?
4) Repeat Exercise 1 for the Matlab script given in Section 3.5.10. Observe and interpret the
impact.
3.8 Exercises for Chapter 3
5) Repeat Exercise 2 for the Matlab script given in Section 3.5.10. Observe and interpret the
impact.
6) Repeat Exercise 3 for the Matlab script given in Section 3.5.10. Observe and interpret the
impact.
7) Repeat Exercise 1 for the Python script given in Section 3.6.1 for bike rental ANN. Observe and
interpret the impact.
8) Repeat Exercise 2 for the Python script given in Section 3.6.1 for bike rental ANN. Observe and
interpret the impact.
9) Repeat Exercise 3 for the Python script given in Section 3.6.1 for bike rental ANN. Observe and
interpret the impact.
10) Repeat Exercise 1 for the Matlab script given in Section 3.6.2 for bike rental ANN. Observe and
interpret the impact.
11) Repeat Exercise 2 for the Matlab script given in Section 3.6.2 for bike rental ANN. Observe and
interpret the impact.
12) Repeat Exercise 3 for the Matlab script given in Section 3.6.2 for bike rental ANN. Observe and
interpret the impact.
49
51
4
ANNs for Financial Stock Trading
4.1 Introduction
The use of computer programs to conduct transactions involving financial securities such as stocks
and bonds is referred to as algorithmic trading. Within this realm, decisions to buy, sell or
hold a diverse set of securities are performed in either autonomous or semi-autonomous fashion,
depending on how comfortable a user is in surrendering control to algorithms.
The clearest advantage of algorithmic trading is its ability to make trading decisions by taking
into account a large number of factors, including analysis of many years of historical trading data;
outside sources, such as media reports, speeches given by leaders and daily events; and many
other inputs influencing trading decisions. Traditional stock traders, no matter how intelligent
and experienced they may be, cannot handle such large quantities of information optimally. In
current financial markets, over 60% of the traded volume utilizes some form of algorithmic trading mechanism. This volume is expected to increase as AI-based tools developed for algorithmic
trading become more sophisticated over time.
4.2 Programs that Buy and Sell Stocks
Finance applications played an important role in the development of various AI methods. The
trading rooms in the New York Stock Exchange from the “old days” have been replaced with the
display screens of remote computers (Figure 4.1).
So-called quants are designers of quantitative analysis and trading algorithms. They are typically
engineers and physicists with mathematical and algorithm backgrounds combined with finance
training.
4.3 Technical Indicators
Algorithmic trading programs typically make their trading decisions by observing so-called technical indicators, which are mathematical formulations derived from historical trading data
such as price and volume. Technical indicators are useful tools to help identify any patterns that
may be hidden in historical data [39]. In this chapter we include the most commonly used indicators, namely momentum, different types of moving averages and Bollinger bands. Although not
described here due to space restrictions, stochastic oscillator [40], rate of change [41], on-balance
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
52
4 ANNs for Financial Stock Trading
(a)
(b)
Figure 4.1 NYSE trading floor: (a) when trading was done by stock brokers (Carol M. Highsmith / The
Library of Congress / Public domain), and (b) after deployment of algorithmic trading by computers (Nicky
Pe / Public Domain Pictures).
volume [42] and Williams %R [43] are among the popular technical indicators. An interested
reader can find many other technical indicators employed in algorithmic trading in [44].
4.3.1 Simple Moving Average
One of the most popular technical indicators, simple moving average (SMA), is defined as
the average price of a stock over a fixed time window of days, weeks or years. SMA is useful in
smoothing out small fluctuations in price by essentially reducing the noise over a selected period
of time. Figure 4.2 (taken from [45]) shows daily stock price movements for S&P 500 stocks and the
corresponding SMA calculated for a 10-month period. As can be seen, fluctuations are eliminated
Figure 4.2
SMA smooths out fluctuations in daily stock movements.
4.3 Technical Indicators
by SMA, and it is much easier to detect trends over time on an SMA curve. Calculation of an n-day
SMA on day t is defined as
1 ∑
price(t − i),
n i=0
n−1
EMA(t) =
where the price of a stock is averaged over the last n days. For example, a 30-day SMA of a stock for
a given day t is the sum of the closing prices from day t to day (t − 30) divided by 30. The SMA can
be computed for any of the financial features of a security (e.g., SMA of opening, closing, maximum
or minimum price or daily volume).
4.3.2 Momentum
The rate of change in a financial asset feature (e.g., price or volume) over a given time interval is
defined as the momentum. This metric is used to determine the strength of a trend for a given stock
over a selected time period. The momentum of two separate points in time is the slope of a line that
connects them. Calculation of momentum on day t over a period of n days is defined as:
M(t) =
price(t)
,
price(t − n)
where M(t) is the momentum at day t. Selecting the most appropriate period of n days to compute
momentum for a given financial asset is an important decision for a financial professional since it
may give different trends for short, medium or large intervals.
A 10-day momentum is shown for the S&P 500 Index in Figure 4.3 (taken from [46]). If the most
recent closing price of the index is more than the closing price 10 trading days ago (i.e., M(t) > 1),
a positive number is plotted above the zero line. Similarly, if the latest closing price is lower than
what it was 10 days ago (i.e., M(t) < 1), a negative measurement is plotted below the zero line. The
zero line represents the index that does not exhibit a trend.
Figure 4.3
Momentum of price movements of S&P 500 Index (taken from [46]).
53
54
4 ANNs for Financial Stock Trading
4.3.3 Exponential Moving Average
Similar to an SMA, the exponential moving average (EMA) is the average price of a stock
over a fixed window of days such that the most recent prices are weighted more than the older ones
in the calculation [47]. To recursively compute an n-day EMA on day t, a percentage of the price
for a day is added to the EMA of the previous day for n days as follows:
(
)
2
2
EMA(t) = price(t) ⋅
− EMA(t − 1) ⋅ 1 −
,
(n + 1)
(n + 1)
where 2∕(n + 1) is called the cooling period. In a period of n days, when calculating the EMA
for the earliest day (let us call it t0 ), the SMA for day t0 is used:
EMA(t0 ) = SMA(t0 ).
The result of the EMA is similar to that of the SMA, except that the older days have less impact on
the outcome. Thus, in theory, the EMA will have less delay in reflecting the latest stock movements
compared to the SMA (Figure 4.4) – since the EMA applies more weight to current data, it follows
prices more closely than a corresponding SMA. However, one has to be aware that the EMA may be
overly sensitive to price changes compared to a comparable SMA. During trading, if the EMA rises
when prices go below the EMA, this may trigger a buy signal (i.e., an indication generated by an
algorithmic trading system that a user should buy this particular stock). Similarly, if the EMA falls
when prices are above the EMA, a sell signal may be generated.
4.3.4 Bollinger Bands
A pair of curves calculated for a given stock based on two standard deviations above and below the
SMA are referred to as Bollinger bands (BBs) [48]. The main idea for using BBs is that the
price for a non-volatile stock will typically stay within the limits of the BBs. If an asset price moves
Figure 4.4 EMA is more sensitive to price changes than SMA, as shown here for the price of a fictitious
financial security for an 8-month period (taken from [47] / FMR LLC).
4.4 A Simple Algorithmic Trading Policy
Figure 4.5
FMR LLC).
Bolinger bands for a fictitious stock computed using 20-day SMA and 2 𝜎 (taken from [49] /
out of the BBs, it is expected to return back into the BB range. This expectation may be a useful
guide in dictating buy or sell decisions for a given stock.
For example, if an asset price goes above the BB, this does not necessarily indicate a buy signal
since it is expected that it will soon return into its BB. Similarly, a stock price falling below its BB may
not trigger and automatic a sell signal since it is expected to rise up into its BB again (Figure 4.5).
For a given day t, BB is computed as follows:
BB(t) = ±
price(t) − SMA(t)
2 ⋅ 𝜎(t)
where 𝜎 stands for the standard deviation of price changes during the same period of n days. Typical
values for a short-term BB is a 10-day SMA and a band of (±1.5 ⋅ 𝜎), whereas a long-term BB uses
a 20-day SMA and a band of (±2.5 ⋅ 𝜎) [49].
4.4 A Simple Algorithmic Trading Policy
Technical indicators can be used in determining an algorithmic trading policy, by which
decisions are made to buy, sell or hold a given stock. The first step of employing technical indicators is to define their parameters, which are referred to as hyper-parameters. For M, the
hyper-parameter is the time range used to compute its slope, whereas for the SMA and EMA, it is
the time range over which the SMA and EMA are to be calculated. For BBs, the hyper-parameters
are the time range for the SMA and the number of standard deviations the bands are away from
the SMA. Selection of hyper-parameter values to be used in technical indicators is a formidable and
non-trivial challenge in algorithmic trading.
An overly simplified example of using the SMA in algorithmic trading can be constructed as
follows. For a given stock, whenever a short-term SMA (e.g., 50 days) crosses above a long-term
SMA (e.g., 200 days), a price increase trend is detected. With the expectation that this
55
56
4 ANNs for Financial Stock Trading
trend will continue, a signal to buy this stock may be generated. If the price keeps increasing, a
profit will be made. Similarly, if a short-term SMA crosses below a long-term SMA, a down trend
is detected and the stock will lose value in in the coming days and, hence, should be sold before
its price gets even lower. This way, a loss is prevented.
It is not obvious how to choose the period over which an SMA is calculated. It requires experience
and knowledge of the specific financial sector in which the stock is traded. If a relatively large n is
selected, the SMA makes the price curve smoother and, hence, it becomes easier to extract information from it since the noise and outliers are eliminated from the price curve. However, the SMA over
larger periods may create a delay between the price movement and SMA curves, which may lessen
the ability to detect recent price movements for traders. On the other hand, a short-period SMA can
respond more quickly to price changes, but with the risk of being more volatile (i.e., a measure that
fluctuates in value more than others).
As an example, consider the two separate SMA curves (for 50 and 200 days) displayed in Figure 4.6
for Bitcoin. One can form a policy called moving average crossover [50] such that when a
short-term SMA crosses over a long-term SMA from beneath, it indicates a potential buy signal.
This means that the short-term SMA is showing a value gain, whereas the long-term SMA has not
responded yet. Similarly, a cross from above indicates a sell signal for a stock (i.e., an indication
from an algorithmic trading system to a user to sell the stock), which implies that the short-term
SMA has detected a loss of value before the long-term SMA. This crossover strategy can be also used
with the EMA [50].
In other applications, if M for a given stock has a relatively large positive value, one may conclude
that the price will keep rising (i.e., a buy signal). Similarly, a relatively large negative value may be a
sign that its value will keep decreasing, implying a sell signal. When using BBs, if the current price
goes above the upper band, it is usually considered a potential sell signal. Likewise, if the current
price goes below the lower band, it is considered to be a potential buy signal. The assumption in
Figure 4.6
50-day and 200-day SMA price curves for Bitcoin (taken from [50] / MYSMP, LLC).
4.5 A Simple ANN for Algorithmic Stock Trading
this approach is that the price will stay within the lower and upper bands of the BBs and that, if the
price goes outside of the BBs, it is expected that it will return into the band shortly. This assumption
implies that if the price is above the upper BB, one must sell the stock since it will soon get back to
the band by dropping its price. Similarly, if the price is below the lower BB, it is expected to come
back into the BB by increasing its price and, hence, a buy signal may be generated.
4.5 A Simple ANN for Algorithmic Stock Trading
ANNs are perfect candidates for predicting stock prices based on historical data recorded during
trading hours over long periods of time (typically in the order of years or decades). Recorded trading
information includes many useful metrics, such as daily opening and closing prices, minimum and
maximum buy and sell prices, and volume, recorded over all functioning financial markets. This
information is referred to as tick data since it was recorded using ticker tapes in the old days of
trading in the early twentieth century using wired connections between stock exchange buildings
and trading houses [51]. Tick data also includes timestamp information for each bid (i.e., the
highest price a buyer is willing to pay for an asset) and ask (i.e., the lowest price a seller is willing
to accept), typically for every minute of a trading day.
In utilizing an ANN for algorithmic trading, a popular approach is to employ a set of technical
indicators computed up to a selected day t as the ANN inputs, whereas output(s) may be selected
as prediction(s) about this financial asset for the following day t + 1 (or the next hour if computed
hourly, etc.). During training, the selected technical indicators are computed for day t as inputs
with a corresponding sample output computed from the tick data. For example, if the ANN output
is selected as percentage change in stock price, the sample output for day t is x percent if the closing
price from day t to day t + 1 was recorded as an increase by x percent. Similarly, if the stock lost its
value by x percent at day t + 1, the output for this sample will be −x percent. Training continues
with the samples obtained from historical tick data recorded for a selected period of several years
or decades. Once the training is over, the ANN is ready to be used for predictions to buy, sell or hold
this stock until a new model is trained in the future.
4.5.1 ANN Inputs and Outputs
In this section we introduce a simplistic ANN to be used for our algorithmic trading example, where
the inputs are a set of technical indicators calculated for user-defined periods (i.e., hyper-parameter
values), and the ANN output is the predicted movement of the price. For this example, we utilize
four technical indicators as ANN inputs, namely, M, SMA, EMA and BB, computed for the closing
price of a given stock (or group of stocks) up to day t. We define prediction of percentage change in
stock price on day t + 1 as the output of the ANN. After an architecture is defined (i.e., the number
of hidden layers and the number of neurons at each layer, activation function assignments,
and so on), the ANN is trained using technical indicators computed based on the historical
tick data.
Typically, a professional stock trader makes the final policy decisions with the help of the
predictions generated by the ANN. For example, as discussed in Section 4.4, if a decrease in price is
predicted, it may trigger a sell signal, whereas a predicted price increase may suggest a buy signal.
In real-life situations, however, a trader may also have other inputs to consider which are not
defined as ANN inputs (e.g., upcoming elections, economic strength, state of the housing market,
57
58
4 ANNs for Financial Stock Trading
unemployment rate, natural disasters) before making policy decisions. For such cases, the ANN
provides enhanced clarity through its predictions based on various technical indicators.
Back-testing is a method that helps in determining an optimal combination of technical
indicators and hyper-parameters for an ANN. Suppose a trader wishes to use the tick data covering
five years of trading information. The ANN designer may choose the data from the first four years
to train and the data from the last year to test whether the trained ANN performance is acceptable.
In this case, once the ANN weights are updated properly over a chosen number of epochs with
the data from years 1 through 4, performance of the trained ANN is tested using the data from
year 5. This is called back-testing, which measures the accuracy of the ANN predictions as if it were
executing algorithmic trading instructions during year 5. If back-testing results are not satisfactory,
the designers modify ANN components until an acceptable combination of architecture, inputs
with proper hyper-parameter values, meaningful outputs, number of epochs to train over and other
similar aspects are determined. Dividing the historical tick data between training and back-testing
is a trade-off that the designers must tackle for the specific market conditions under which they
are operating. For some applications, the ANN is trained daily, whereas for other markets ANNs
trained a few times a year are preferred. In summary, to engage an ANN for algorithmic trading, at
minimum, the following steps should be performed:
Select technical indicators and/or other variables to be used as ANN inputs (e.g., M, SMA, EMA
and BB, interest rates, unemployment rate).
● Define hyper-parameter values for the selected technical indicators (e.g., SMA(5), EMA(20),
the latest interest rates averaged over last two quarters).
● Define an ANN architecture (i.e., number of layers, number of neurons per layer, connections
between layers, activation functions, etc.).
● Split the historical data for training and back-testing.
● Determine or compute the ANN inputs and corresponding outputs for each training day using
the training portion of the historical data.
● Train the ANN by applying the inputs, computing the errors in its output and updating the
weights for a pre-determined number of epochs.
● Using the back-testing portion of historical data, measure the ANN’s performance.
● Based on the back-testing results, if needed, modify the ANN architecture as well as the selection of ANN inputs and outputs and repeat the above steps until a satisfactory performance is
obtained.
● Once the ANN is trained properly, it is ready to be used in the field: new inputs are provided
using the latest information to obtain predictions.
●
4.5.2 ANN Architecture
For the simplistic ANN architecture used in this chapter, as shown in Figure 4.7, we define four
inputs, namely M, SMA, EMA and BB, and one output as the percentage change in price for the
following day. Note that W and B in the figure are the weight matrices and the bias vector defined
in Chapter 3. In this simplistic ANN, there are two hidden layers with four and three neurons,
respectively. The ANN is fully connected. A sample of the tick data used for training this ANN is
shown in Figure 4.8, which contains daily opening, highest, lowest, closing and adjusted closing
prices (i.e., the closing price re-computed after the trading day ends to reflect actions on the stock
price, such as paying dividends, stock splitting, etc. [51]) and volume (i.e., the number of units sold
and bought) for the selected example of Microsoft stock.
4.6 Python Script for Stock Trading Using an ANN
Figure 4.7
Simplistic ANN with two hidden layers for stock price prediction.
Figure 4.8
Sample tick data for Microsoft stock.
4.6 Python Script for Stock Trading Using an ANN
A sample Python script implementing the ANN shown in Figure 4.7 is presented below. In line 22,
the tick data file (called MSFT.txt) is defined, whose closing price is saved in a one-dimensional
array called df in line 26. Short and long windows to be used for SMA and EMA computations
later are defined as 5 and 25 days in lines 28 and 29, respectively. In line 33, SMA is computed and
added to df as a new column as follows:
df['SMA'] = df.rolling(short_window).mean()
where rolling means that the moving window for SMA starts from the current entry and
includes the last five days (defined by short_window) for each day. Similarly, EMA will be
59
60
4 ANNs for Financial Stock Trading
computed and added to df as a new colums as defined in line 37 as follows:
df['EMA'] = df.ewm(span=long_window).mean()
where ewm means EMA using 25 days (defined by long_window) for each day. Similarly, M
and BB are defined in lines 40 and 47, respectively. The ANN output, which is the percentage
change on Close, is defined in line 55. The array is shifted by one such that the percent change
for day i + 1 is aligned with the technical indicators for day i. This way, the ANN weights will
be trained to match the percent change of the following day, which will constitute a prediction
for the next day. In lines 68 and 69, the ANN inputs and output are defined as arrays X and Y,
respectively.
The ANN architecture is defined in lines 72–82. In line 72, an object model is created to hold
the ANN. In line 78, a hidden layer is defined to accept four inputs:
model.add(keras.layers.Dense(4,activation = 'relu',input_shape = (4,)))
where Dense means all neurons are connected to the previous layer (i.e., the ANN inputs). A
second hidden layer is added to the ANN in line 88 with 3 neurons:
model.add(keras.layers.Dense(3,activation = 'relu'))
An output layer using a linear activation is added in line 82 as follows:
model.add(keras.layers.Dense(1,activation = 'linear'))
where negative values of output are not filtered by setting activation as linear. The training
parameters are set in line 86 as
model.compile(optimizer = 'adam',loss = 'mean_square_error')
where adam uses a gradient descent algorithm whose convergence will be measured by the
mean_square_error method. In line 88, the ANN is trained:
model.fit(X,Y,epochs=200)
Once the ANN is trained, it is ready to make predictions on percentage change in price for
the next day. In lines 102–106, SMA, EMA, M and BB are computed for a new day and saved
in an array called latest_inputs, which is then applied to the trained ANN to generate
a prediction:
prediction = model.predict(latest_inputs)[0][0]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# The City College of New York, City University of New York
# Written by Ricardo Valdez
# August, 2020
# Algorithmic trading of financial stocks using ANNs
import numpy as np
import pandas as pd
import time
from tensorflow import keras
def print_weights(weights):
# weights = model.get_weights();
print('\n******* WEIGHTS OF ANN *******\n')
for i in range(int(len(weights)/2)):
print('Weights W%d:\n' %(i), weights[i*2])
4.6 Python Script for Stock Trading Using an ANN
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
print('Bias b%d:\n' %(i), weights[(i*2)+1])
#% DATA EXTRACTION
historical_data_file = 'MSFT.csv'
## load the historical stock data
df = pd.read_csv(historical_data_file) # (df stands for data frame)
# ## get the date of the latest trading day
# latest_trading_day_date = df['Date'].iloc[-1]
## keep only the closing prices
df = df[['Close']]
short_window = 5
long_window = 25
## calculate the Simple Moving Average and add it to a new column in df;
## rolling means moving window starting from the current entry and
## the last 5 days
df['SMA'] = df.rolling(short_window).mean()
## calculate the Exponential Moving Average and add it
## to a new column in df;
## ewm means exponential moving average
df['EMA'] = df['Close'].ewm(span=long_window).mean()
## calculate the Momentum and add it to a new column in df;
## Momentum is defined as close [today] / close [5 days ago]
df['Momentum'] = df['Close'] / df['Close'].shift(short_window)
## calculate the standard deviation of SMA over a rolling window
## ddof means degree of freedom, set it to zero to get the population std
df['STD'] = df['Close'].rolling(short_window).std(ddof=0)
## calculate the Bollinger Band for each day
## BB > 1 means the closing price is above the upper band
## BB > -1 means the closing price is below the lower band
df['BB'] = (df['Close'] - df['SMA']) / (2 * df['STD'])
## calculate the percent change of closing price for each day
## shift(1) means yesterday (applies to each element of df)
increase = df['Close'] - df['Close'].shift(1)
df['Percent_Change'] = (increase / df['Close'].shift(1))*100
## shift percent change to the previous day so that it now represents the
## percent change for the next day (shift df elements up by 1 using -1)
df['Percent_Change'] = df['Percent_Change'].shift(-1)
#% ANN TRAINING
## start training ANN using the df file contains historical data and
## information that was just populated above
print('\n\n***** NOW START TRAINING ANN USING', historical_data_file,'*****')
time.sleep(3)
## remove rows with invalid inputs (i.e., nan) and
## create input and output arrays for ANN
## starting from day 25 to te end, but excluding the end (due to -1)
## we will predict the precent change of the last day recorded in csv (today)
X = np.array(df[long_window:-1][['SMA','EMA','Momentum','BB']])
Y = np.array(df[long_window:-1]['Percent_Change'])
## create a model for the ANN
model = keras.Sequential()
# Create ANN with 3 inputs + 2 hidden layers + 1 output layer
## first hidden layer that accepts 4 input features (SMA, EMA, Momentum, BB);
## the hidden layer will have 4 neurons;
## dense means every neuron in the layer connects to every neuron in the
## previous layer;
model.add(keras.layers.Dense(4, activation='relu', input_shape=(4,)))
## add another hidden layer with 3 neurons to the ANN
model.add(keras.layers.Dense(3, activation='relu'))
## add an output layer with a single output (percent change)
model.add(keras.layers.Dense(1, activation='linear'))
## set the optimization algorithm used for minimizing loss function
## use gradient descent (adam) to minimize error (loss)
model.compile(optimizer='adam', loss='mean_squared_error')
## train the ANN model using 200 iterations
61
62
4 ANNs for Financial Stock Trading
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
model.fit(X, Y, epochs=200)
## training with more iterations will yield better results
## build different ANN configurations for better results
## use different activation functions for better results
## use different optimizers adam or SGD (stochastic gradient descent)
## model.fit(X, Y, epochs=2000)
weights = model.get_weights()
print_weights(weights)
print('\n\n********** ANN training complete **********\n\n')
#% ANN PREDICTION
## insert the inputs for the latest trading day into an array
latest_SMA = df.iloc[-1]['SMA']
latest_EMA = df.iloc[-1]['EMA']
latest_Momentum = df.iloc[-1]['Momentum']
latest_BB = df.iloc[-1]['BB']
latest_inputs = np.array([[latest_SMA, latest_EMA,latest_Momentum,latest_BB]])
prediction = model.predict(latest_inputs)[0,0]
print('\n***************************************')
print('ANN Predicted Next Day Stock Movement: %+.2f%%' % (prediction))
print('***************************************')
A sample run is shown in Figure 4.9, where the weights of the ANN and the prediction of price
for the following day are printed. Due to the non-deterministic nature of ANNs, outcomes for the
same data may differ for successive executions of the same script. Generating consistent results
requires proper selection of technical indicators and ANN parameters for a given stock.
Figure 4.9
Sample output of the Python script.
4.7 Matlab Script for Stock Trading Using an ANN
4.7 Matlab Script for Stock Trading Using an ANN
A sample Matlab script for the ANN shown in Figure 4.7 is presented below. The structure and
variables of the Matlab scripts here are kept similar to the ones in Python given in Section 4.6 for
ease of comparison by the reader.
In line 18, tick data is read into bf df array. Short and long window sizes are defined as 5 and
25 in lines 19 and 20, respectively. In lines 25, 29, 32 and 40, technical indicators of SMA, EMA, M
and BB are defined, respectively. In line 47, the percentage change in price is computed by dividing
the daily price difference by the closing price of the previous day. In line 48, they are shifted by one
day to align the increase for the following day as the present day. Input vector X, which contains
the daily values of the technical indicators computed above, is defined in line 57 and the output
vector of Y, which contains the percent price changes, in line 58.
The ANN architecture is defined in lines 63–70 such that the ANN has two hidden layers, with
four and three neurons, and an output layer with one neuron. All layers are fully connected, meaning that all previous layer inputs are connected to all neurons at current layer as shown in Figure 4.7.
In lines 72 to 76, options for ANN training are defined: the number of epochs is 200, batch size is
10 (i.e., the samples are handled 10 at a time to reduce matrix multiplication complexity) and a
learning rate of 𝛼 = 0.01. In line 77, the ANN is trained and saved in an object named model as:
model=trainNetwork(X,Y,layers,options);
In line 81, SMA, EMA, M and BB are computed for a new day and saved in an array called
latest_inputs, which is then applied to the trained ANN as an input by engaging the library
function named predict:
prediction=predict(model,latest_inputs)
where the predicted percentage change of price for the new day is stored in the variable
prediction.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
% The City College of New York, City University of New York
% Written by Olga Chsherbakova
% Date: November, 2023
% ANN for algorithmic trading of financial assets
clc;
clear;
% start training ANN using the df file contains historical data and
% information that was populated below
fprintf('********* NOW START TRAINING ANN USING MSFT.csv *********')
% load the historical stock data
historical_data_file = 'MSFT.csv';
% df stands for data frame
df = readtable(historical_data_file, 'PreserveVariableNames', true);
% keep only the closing prices
df = df(:, {'Close'});
short_window = 5;
long_window = 25;
% calculate the Simple Moving Average and add it to a new column in df;
% rolling means moving window starting from the current entry and
% the last 5 days (open df in Variable explorer ad see the values)
df.SMA = movmean(df.Close, short_window);
% calculate the Exponential Moving Average and add it
% to a new column in df;
% ewm means exponential moving average
df.EMA = movavg(df.Close, 'exponential', long_window);
% calculate the Momentum and add it to a new column in df;
% Momentum is defined as close [today] / close [5 days ago]
63
64
4 ANNs for Financial Stock Trading
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
df.Momentum = df.Close./ lagmatrix(df.Close, short_window);
std_vals = movstd(df.Close, short_window, 'Endpoints', 'discard');
% calculate the standard deviation of SMA over a rolling window
% set degree of freedom to zero to get the population std
df.STD = padarray(std_vals, [short_window-1 0], NaN, 'pre');
% calculate the Bollinger Band for each day
% BB > 1 means the closing price is above the upper band
% BB < -1 means the closing price is below the lower band
df.BB = (df.Close - df.SMA)./ (2 * df.STD);
% calculate the percent change of closing price for each day
% shift(1) means yesterday (applies to each element of df)
increase = df.Close - lagmatrix(df.Close, 1);
% shift percent change to the previous day so that it now represents the
% percent change for the next day (shift df elements up by 1 using -1)
df.Percent_Change = (increase./ lagmatrix(df.Close, 1)) * 100;
df.Percent_Change = lagmatrix(df.Percent_Change, -1);
% ANN TRAINING
% remove rows with invalid inputs (i.e., nan) and
df = rmmissing(df);
% Create input and output arrays for ANN
% starting from day 25 to te end, but excluding the end (due to -1)
% predict the precent change of the last day recorded in csv (today)
X = table2array(df(long_window:end-1, {'SMA', 'EMA', 'Momentum', 'BB'}));
Y = table2array(df(long_window:end-1, {'Percent_Change'}));
% create a model for the ANN
% Create ANN with 4 inputs + 2 hidden layers + 1 output layer
% first hidden layer that accepts 4 indicators (SMA, EMA, Momentum, BB);
layers = [
featureInputLayer(4)
fullyConnectedLayer(4)
reluLayer
fullyConnectedLayer(3)
reluLayer
fullyConnectedLayer(1)
regressionLayer];
% Plot
options = trainingOptions('adam', ...
'MaxEpochs', 200, 'MiniBatchSize', 10, 'InitialLearnRate', 0.01, ...
'LearnRateSchedule', 'piecewise', 'LearnRateDropPeriod', 125, ...
'LearnRateDropFactor', 0.2, 'Verbose', false, 'Plots', ...
'training-progress');
model = trainNetwork(X, Y, layers, options);
% ANN PREDICTION
% insert the inputs for the latest trading day into an array
latest_inputs = table2array(df(end, {'SMA', 'EMA', 'Momentum', 'BB'}));
prediction = predict(model, latest_inputs);
% Print results
fprintf('\n********** ANN training complete **********')
fprintf('\n***************************************\n');
fprintf('ANN Predicted Next Day Stock Movement: %+0.2f%%\n', prediction);
fprintf('***************************************\n');
Figure 4.10 Sample output generated by the Matlab script presented above, showing a 0.07% increase
predicted for Microsoft stock.
4.9 Exercises for Chapter 4
A sample output is shown in Figure 4.10, in which a 0.07% increase is predicted for Microsoft
stock. Due to the non-deterministic nature of ANNs and differences in library functions, the outcomes for the same data differs for the Python and Matlab scripts.
4.8 Concluding Remarks
ANNs are excellent candidates for implementing algorithmic trading of financial assets based on
historical tick data. An underlying assumption is that financial tick data may have certain characteristics that can be captured by properly designed and trained ANNs, giving an advantage to
traders, who would otherwise be forced to use their own personal experience and skills to extract
or infer such information. A known problem in ANN training is inadvertently over-fitting
ANN weights to tick data, where back-testing results perform well for historical data and not for
real market situations. Avoiding over-fitting in ANN training is currently an open research topic.
In this chapter we introduce a simplistic ANN model with four popular financial technical indicators as inputs generating percentage change of price as its output. This ANN model is fully
connected and contains two hidden layers. Using Python and Matlab scripts, we illustrate how
to implement the simplistic ANN model together with its hidden layers, activation functions, error
calculation method on output and the number of epochs to be used for algorithmic trading of financial stocks. Although library functions available in Python and Matlab make such implementations
trivial in principle, poor execution speeds and restricted control over ANN parameters mean that
professional-level applications require custom software that is developed, monitored and maintained by expert programmers and finance experts.
Although modern computer systems can provide considerable computational power, one has
to be cautious in selecting the ANN architecture and its inputs and outputs, as well as devising
trading policies based on such information. It is not clear how best to select technical indicators
used, hyper-parameters, number of hidden layers and activation functions to provide meaningful
guidance to professional stock traders. It seems that current algorithmic trading applications
require experience both in computing and finance: the skill-set required for selection of technical
indicators requires expertise in the finance industry, whereas constructing an effective ANN
architecture is typically a task for data scientists.
4.9 Exercises for Chapter 4
In the exercises below, the following technical indicators are used:
SMA: Simple moving average
EMA: Exponential moving average
M: Momentum
BBs: Bollinger bands
SO: Stochastic oscillator
ROC: Rate of change
OBV: On balance volume
WR: Williams %R
1) Using the ANN given in Figure 4.7, predict the next-day closing price movement for Google
stocks. Use the following six technical indicators as inputs for your ANN: SMA, EMA, M, BB,
SO and WR.
2) Using the ANN given in Figure 4.7, predict the next-day closing price movement for Tesla
stocks. Use the following six technical indicators as inputs for your ANN: SMA, EMA, M, BB,
SO and ROC.
65
66
4 ANNs for Financial Stock Trading
3) Using the ANN given in Figure 4.7, predict the next-day closing price movement for Apple
stocks. Use the following six technical indicators as inputs for your ANN: SMA, EMA, M, BB,
SO and OBV.
4) Using the ANN given in Figure 4.7, predict the next-day closing price movement for Zoom
stocks. Use the following six technical indicators as inputs for your ANN: SMA, EMA, M, BB,
WR and OBV.
5) Using the ANN given in Figure 4.7, predict the next-day closing price movement for Amazon
stocks. Use the following six technical indicators as inputs for your ANN: SMA, EMA, BB, WR
and OBV.
6) Using the ANN given in Figure 4.7, predict the next-day closing price movement for Netflix
stocks. Use the following six technical indicators as inputs for your ANN: SMA, EMA, M, BB,
ROC and OBV.
7) Repeat Exercise 1 using an ANN with three hidden layers. Why are the results different?
8) Repeat Exercise 1 using an ANN by increasing the number of epochs and changing its
activation functions. Why are the results different?
9) If you repeat Exercise 1 by using three technical indicators instead of six, what differences do
you expect in the ANN predictions?
10) If you repeat Exercise 1 by using nine technical indicators instead of six, what differences do
you expect in the ANN predictions?
67
5
ANNs for Alzheimer’s Disease Prognosis
5.1 Introduction
In this chapter we discuss how machine learning techniques can be exploited for solving otherwise
formidable and sometimes intractable problems in life sciences. As a case study, we show how an
ANN can be employed for prediction of Alzheimer’s disease (AD) prognosis based on personalized
biological information. We also present sample Python and Matlab script implementations of such
an ANN, trained by field data for biomarkers collected from AD patients over time.
5.2 Alzheimer’s Disease
Alzheimer’s disease (AD) is an incurable progressive neurodegenerative disorder caused by
destruction of neurons within the brain, with typical symptoms of cognitive decline, memory
loss, behavioural changes and functional impairment of an individual. As the disease progresses,
individuals with AD often experience worsening symptoms and a decline in overall health.
Everyone who develops Alzheimer’s dementia first experiences mild cognitive impairment (MCI). Among those with MCI, about 15% develop dementia after two years and approximately one-third within five years. However, some individuals with MCI do not have additional
cognitive decline or revert to normal cognition. Among population-based studies, a systematic
review and meta-analysis reported a reversion rate of 26% [52]. Therefore, it is vitally important
to develop tools to predict whether a patient with MCI will decline (i.e. progressive MCI) or remain
stable (i.e. stable MCI). Determining disease trajectories for individuals diagnosed with MCI has
major societal impact through personalized interventions for patients, their families, caregivers
and the insurance industry.
AD is associated with alterations in gene expression patterns in the brain. These changes affect
various biological processes, including inflammation, synaptic functions and neuronal survival.
The morbidity rate after an AD diagnosis refers to the frequency and incidence of health
complications, disabilities and adverse events experienced by patients. It depends on several factors, including the age at diagnosis, overall health of an individual, access to healthcare and presence of other underlying medical conditions. AD is typically a chronic condition that leads to
significant disability and dependence on others for daily activities as it advances. However, it is
important to note that AD is a highly individualized disease, and the rate and progression of morbidity vary among individuals. Adequate support, management of symptoms, access to healthcare
and appropriate care-giving can help mitigate some of the morbidities associated with AD and
improve patient quality of life despite the challenges posed by the disease [53].
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
68
5 ANNs for Alzheimer’s Disease Prognosis
Measures to assess an individual’s cognitive abilities, including memory, attention, language and
executive functions, are called cognitive scores. Standardized tests are often used to evaluate
the severity of cognitive decline in patients. Mini Mental State Exam (MMSE) scores are
popular for assessing cognitive functions of a patient over a period of time [54]. In MMSE, there
are 30 questions that a patient is asked (starting from as simple a question as the names of siblings).
Another important biomarker is gene expressions, which can act as strong indicators of
AD, as alteration of certain genes can influence disease progression [55]. Gene expression is a
process by which a gene gets turned on in a cell to make ribonucleic acid (RNA) and proteins.
Gene expression may be measured by looking at what the protein does in a cell. The relationship between gene expressions and cognitive scores is complex and multifaceted. Gene expression
changes can contribute to the development and progression of AD, influencing the pathological
processes involved. These changes can ultimately impact cognitive abilities, leading to declines in
scores. The Alzheimer’s disease Neuroimaging Initiative (ADNI) recorded the
scores of each patient every week for up to two years, along with their gene expressions [56]. The
ADNI database contains data from a wide breadth of participants, including individuals with AD,
MCI and no cognitive issues. Its primary objective is to identify and validate biomarkers for AD
progression, develop improved diagnostic criteria and accelerate the development of new therapies. ADNI data includes detailed clinical assessments, neuropsychological test results, medical
histories, genetic information, brain imaging data and cerebrospinal fluid biomarker measurements. The ADNI project has had a significant impact on the field of AD research and has resulted
in numerous scientific publications, collaborations and insights into the disease.
It has been an ongoing research effort for utilizing ANNs for AD prognosis using various
biomarkers as inputs to generate a prediction of rate of decline in cognitive abilities. For example,
imaging modalities, clinical measures and cognitive status markers are used in [57]. MRI images
of the brain sourced from multiple databases for healthy individuals and those diagnosed with
AD to form similarities via ANNs for disease progression [58]. In [59], the researchers focus on
biomarkers that may influence AD progression, such as cerebrospinal fluid, plasma biomarkers,
oxidative stress, metal ions, vascular disorders, protein dysfunctions and alterations in the
mitochondrial populations, along with markers obtained from amyloid imaging. In another
study [60], researchers employ MRI information for hippocampal images and gene expressions
together with MMSE scores to predict AD progression.
5.3 A Simple ANN for AD Prognosis
Let us now define a simple ANN for AD prognosis whose inputs are the expression values of
AD-related genes collected from patients diagnosed with AD and the output is M-coefficients
(i.e. MMSE score loss per week), which are direct indicators of cognitive decline. If the
M-coefficients are small, the decline will be slower, whereas a larger value is indicative of fast
decline in cognitive abilities. Once the ANN is trained, it is expected that a relationship with an
AD patient’s gene expressions (for AD related genes) and the disease progress can be formulated
through the weights and biases of the trained ANN. When a new AD patient is identified, the gene
expressions for those 63 genes will be obtained from the patient and fed into the trained ANN to
generate a personalized prediction of AD prognosis specifically computed for that patient.
Our simple sample data set consisted of 63 gene expressions taken from 215 patients present
in the ADNI database. M-coefficients, also obtained from the ADNI database, were normalized
between 0 and 1 to lessen the possible negative impact of outliers during training. The inputs for
the ANN are selected as the gene expressions of the patients. A few of the gene expressions for a
5.3 A Simple ANN for AD Prognosis
Figure 5.1 Gene expressions are selected as ANN inputs; a few of them are shown here for a small sample
of patients (data taken from ADNI database [56]).
small subset of patients are shown in Figure 5.1. The output of the ANN is selected as the slope
of MMSE score decline, which indicates the rate at which the disease will progress. MMSE score
slopes for the sample set of patients whose gene expressions are given in Figure 5.1 are used as the
ANN outputs, as shown in 5.2.
We defined an ANN architecture with four hidden layers with 120, 60, 15 and 5 neurons
in the first through the fourth hidden layers, respectively, as shown in Figure 5.3 (referred to
as 120-60-15-5-1 architecture). Note that after experimenting with different architectures, this
combination of the number of hidden layers and the number of neurons in each layer happen to
perform the best.
During training, to reduce the influence of possible outliers on the sample data on ANN weight
computations, we employed the so-called leave-one-out-cross-validation (LOOCV)
method to measure the performance of the trained ANN [61]. Suppose there are 100 data samples
(i.e. 100 sets of inputs and corresponding outputs) to be used for training an ANN. In LOOCV, one
data sample out of 100 (let us call it sample 1) is taken out of the data set, and the ANN is trained
using the remaining 99 samples. After the training is complete, ANN performance is measured
by using sample 1 as a test, where inputs of sample 1 are applied to the ANN, the output(s) are
generated by the ANN and an error value is computed. After that, the same process is repeated
taking another sample (e.g. sample 2) out of the 100 samples, and so on. At the end, there are
100 different error values, which indicate the performance of the ANN with respect to the sample
data. In the general case, LOOCV can be conducted by removing k samples out of N, training the
ANN with (N − k) samples and computing the performance for k tests. Selection of k depends on
the application for which the ANN is designed.
When an ANN performs well for a sample data set used in training but fails to perform for new
data elements, it is said that the ANN is over-fitted to its sample data. Over-fitting is especially
observed in ANNs with too many hidden layers and/or too many neurons in hidden layers. This is
due to noise in the sample data that fools the ANN weights as being important information. One of
the methods to avoid over-fitting is engaging regularization techniques and reducing noise
69
70
5 ANNs for Alzheimer’s Disease Prognosis
Figure 5.2 Slope of MMSE scores are selected as the ANN outputs; a sample subset of patient scores are
shown here, which correspond to the gene expressions in Figure 5.1 (data taken from ADNI database [56]).
Figure 5.3 ANN architecture 120-60-15-5-1, where the inputs are the expressions of 63 AD-related genes,
and the output is the prediction for MMSE score weekly decline rate.
influence in weight computation, where the loss function is modified such that weights with small
values play a less important role in error computation. For example, in L1 regularization,
small weights are encouraged to be zero, and, hence, their impact on performance is significantly
reduced [62]. In the ANN implementation presented here, L1 regularization is employed in the
first two hidden layers with 120 and 60 neurons.
5.4 Python Script for AD Prognosis Using an ANN
5.4 Python Script for AD Prognosis Using an ANN
In this section we present a sample Python script implementing the 120-60-15-5-1 architecture
shown in Figure 5.3, whose inputs are the expressions of 63 AD-related genes and the output is the
prediction of weekly decline rate for the MMSE score.
An input file is defined as the source of sample data points in line 14. In lines 16 and 21, input
samples are saved in an vector called X. Similarly, outputs from the samples are saved in Y in line
25. In lines 36 and 37, output values (i.e. decline slopes of MMSE scores) are normalized to avoid
numerical inaccuracies during training. The loop between lines 56 and 245 processes four patients,
which are taken out of sample set during training as part of LOOCV method described above. In
this LOOCV implementation, only four patients are taken out of the 215-patient sample set during
training (one patient at a time for each iteration of the loop), and hence the ANN was trained four
separate times, each with 214 patients. In line 83, an object named model is created to hold the
ANN. In lines 96–100, a hidden layer with 63 inputs and 120 neurons is created. Consecutive hidden
layers with 60, 15 and 5 neurons are created in lines 103, 106 and 109, respectively. The output layer
with a single neuron is created in line 113.
In all hidden layers, the activation function is selected as rectified linear unit (i.e. relu), whereas
the output layer uses linear activation (so as not to disallow negative results in the output). As mentioned earlier, L1 regularizers are employed in the first two hidden layers as a prevention measure
against over-fitting, as they have high numbers of neurons. All hidden layers are defined as dense
for a fully-connected ANN. In line 122, the optimizer option is defined as adam to employ a gradient descent algorithm to update the parameters of the weights/bias based on loss, which is defined
to be measured using the mean absolute error (i.e. the option mae). A learning rate of 𝛼 = 0.01 is
selected. The number of epochs used to train the ANN is set as 500 in line 128.
In line 139 a prediction is generated for the selected gene expressions of the ith individual in
the loop. The prediction is normalized between lines 144 and 151. The error is computed in lines
153–158. Plots for the predictions and for the real sample data are generated in lines 168 to 256.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# The City College of New York, City University of New York
# Written by Grace McGrath and Ishmam Fardin
# July 2023
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler
import warnings
warnings.filterwarnings("ignore") # supresses all warnings
# Personalized prognosis of Alzheimer's disease progpression using ANNs
# Load the training data, Input X
training_X = 'training_data/gene_expressions.csv'
X = pd.read_csv(training_X)
# filters out unncessary information like age, name, etc.
# only focuses on gene expressions
X_in = X.filter(regex='x_at', axis=1)
# print(X_in.head())
# Load the rest of the training data. Output Y
training_Y = 'training_data/mmse_score_slopes.csv'
Y = pd.read_csv(training_Y)
# print(Y.head())
# fit_transform converts all the values which are the slope of cognition
71
72
5 ANNs for Alzheimer’s Disease Prognosis
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
# decline into values that are between 0 and 1
# using MinMaxScaler which does (original value - minimum) divided by
# (maximum - minimum) to get the new value
# this is done to help the algorithm perform better and also leads to outliers
# having less impact
# in reshape(-1,1) the -1 signifies the number of rows in the array and the 1
# represents one column reshapes converts the array into a 2D array
mmse_slopes = np.array(Y["m-coef(mmse-points/week)"]).reshape(-1,1)
Y_scaled = MinMaxScaler().fit_transform(mmse_slopes)
# hold all percent errors for each sample that was left out for LOOCV
percent_slope_errors = []
# hold all mmse score errors
percent_mmse_score_errors = []
# hold all predictions
predictions = []
# hold predictd_mmse_scores
predicted_mmse_scores = []
# hold all actual slopes
actual_slopes = []
# hold actual_mmse_scores
actual_mmse_scores = []
# hold all patient IDs
patient_IDs = []
# Find the percent error for each sample that's left out
# Utilize LOOCV to evaluate performance of model
for i in range(4):
print("\n********************************")
print(f"LEAVING OUT ROW {i} \n")
# initialize a copy of all inputs to a new variable so that original
# variable does not get updated
inputs = X_in.copy()
# initialize a copy of all outputs that were scaled to a new variable so
# that original variable does not get updated
outputs = Y_scaled.copy()
# save the input for test sample that will be used to see how effecient
# the model is
LOOCV_input = inputs.loc[i]
# get the slope of test sample
actual_slope = list(Y["m-coef(mmse-points/week)"])[i]
# drop the row with test sample inputs
inputs.drop(i,inplace=True)
# remove the output of test sample from the output array that will be used
#to train model
outputs = np.delete(outputs, i)
# create a model for the ANN
model = keras.Sequential()
print("Training Started")
# remove / add hashtags to test architectures
# Define and train model architecture 120-60-15-5-1
# add a hidden layer that takes in 63 input features
# (the gene expressions that have x_at)
# hidden layer has 120 neurons
# dense connects all of it's neurons to the neurons in the layer after,
# and the layer before
model.add(keras.layers.Dense(120,
kernel_regularizer='l1',
activation='relu',
input_shape=(63,),)
)
5.4 Python Script for AD Prognosis Using an ANN
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
# add another hidden layer with 60 neurons
model.add(keras.layers.Dense(60,kernel_regularizer='l1', activation='relu'))
# add another hidden layer with 15 neurons
model.add(keras.layers.Dense(15, activation='relu'))
# add another hidden layer with 5 neurons
model.add(keras.layers.Dense(5,activation='relu'))
# add an output layer with one output (the slope of the cognitive decline)
# activation function default to linear
model.add(keras.layers.Dense(1))
# set optimizer to Adam which is a gradient descent method that updates
# the weights to minimize the loss
# set loss to mean_absolute_error which does the average absolute
# difference between actual output and estimated output
# set learning rate to 0.01, learning rate is the rate at which model
# paremeters are updated)
# metrics is utilized to see how the model is performing each epoch
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
loss='mean_absolute_error',
metrics=['Precision']
)
# train the model for 500 epochs
model.fit(inputs, outputs, epochs = 500, verbose=0)
print("Training Finished\n")
gene_expressions = LOOCV_input.to_frame() # converts series to dataframe
# transpose rows/columns so that it is compatible with
# model input configuration
gene_expressions_transposed = gene_expressions.transpose()
# make a prediction based on inputs
prediction_scaled = model.predict(gene_expressions_transposed)
# initialize the current patient ID to a variable
patient_ID = X['patient-id'][i]
patient_IDs.append(patient_ID)
# find the minimum of the original output values
mmse_score_min = Y['m-coef(mmse-points/week)'].min()
# find the maximum of the original output values
mmse_score_max = Y['m-coef(mmse-points/week)'].max()
# normalize the prediction
mmse_max_min_difference = mmse_score_max - mmse_score_min
prediction = mmse_score_min + prediction_scaled*mmse_max_min_difference
# calculate percent error
rel_err = abs(((prediction - actual_slope)/actual_slope)*100)
rel_error = round(float(rel_err[0][0]), 2)
print("\nThe relative error is: {}%".format(rel_error))
percent_slope_errors.append(rel_error) # adds percent error to list
# Round the prediction and actual slope
prediction = round(float(prediction[0][0]), 3)
actual_slope = round(float(actual_slope), 3)
print(f"The predicted MMSE Score slope is: {prediction} mmse-points/week")
print("The actual slope is:", actual_slope)
predictions.append(prediction) # adds prediction to list
actual_slopes.append(actual_slope) # adds actual slope to list
# Generate a plot of the predicted ouptut vs the actual
# baseline is the score that the patient started with
baseline = list(Y["m-baseline(mmse-points)"])[i]
# generate a sequence of evenly spaced numbers between 0 and 52 with
# a total of 7 values
73
74
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
5 ANNs for Alzheimer’s Disease Prognosis
x = np.linspace(0, 52, 7)
# create a figure to hold all the plots, and makes it a white background
# thats 8 by 6 inches
plt.figure(figsize=(8, 6), facecolor='white')
plt.subplots_adjust(top=0.85)
# Adjust top margin for title
# string representation of the equation for actual output with
# baseline initialized to a variable
actual_equation = f"Actual: y = {actual_slope}x + {round(float(baseline),2)}"
# string representation of the equation with actual output with
# baseline initialized to a variable
rounded_baseline = round(float(baseline),2)
prediction_equation = f"Predicted: y = {prediction}x + {rounded_baseline}"
# calculate the minimum y-value for the actual data
actual_min = actual_slope*52+baseline
actual_mmse_scores.append(round(float(actual_min),2))
# calculate the minimum y-value for the predicted data
predicted_min = prediction*52+baseline
predicted_mmse_scores.append(round(float(predicted_min),2))
# record percent_error_mmse_score
percent_mmse_score_errors.append((predicted_min-actual_min)/actual_min)
# Show equation of each line on the graph
# add a text annotation to the plot at the coordinates (2, actual_min)
# The text content is given by actual_equation, and it will be displayed
# in the color 'palevioletred'
# The ha='left' and va='bottom' parameters specify the horizontal
# and vertical alignment of the text, respectively
plt.text(2, actual_min+1, actual_equation, color='palevioletred',
ha='left', va='bottom')
# add a text annotation at the coordinates (2, predicted_min)
# The text content is given by prediction_equation, and it will be displayed
# in the color 'palevioletred'
# The ha='left' and va='bottom' parameters represent horizontal and vertical
# alignment of the text
plt.text(2, predicted_min, prediction_equation, color='skyblue',
ha='left', va='top')
# plot the line of the actual output, sets color of line to palevioletred,
# and for the legend, it adds a label title
plt.plot(x, actual_slope*x + baseline, color='palevioletred',
label ='real MMSE score decline (best fit)')
# plot the line of the predicted output, sets color of line to skyeblue,
# and for the legend, it adds a label title
plt.plot(x, prediction*x+baseline, color='skyblue',
label ='predicted MMSE score decline')
plt.xlabel('Weeks Since First Mini-Mental State Exam') # sets x axis label
plt.ylabel('MMSE Score') # sets y axis label
plt.xlim([0, 52]) # sets x axis range from 0 to 52
# set title
title = "MMSE Score Decline \n Predicted vs Actual \n Patient:"
plt.title(title + patient_ID)
plt.legend() # shows the legend
# save the graph into predictions directory
plt.savefig(f'predictions/{patient_ID}.png')
print("****************************")
print("LOOP DONE \n")
# List the percent error, slopes
5.5 Matlab Script for AD Prognosis Using an ANN
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
for i in range(len(percent_slope_errors)):
print("\nPatient ID:", patient_IDs[i])
print(f'Slope of Cognitive Decline: Actual: {actual_slopes[i]}\
Predicted: {predictions[i]}')
print(f'Percent Slope Error: {percent_slope_errors[i]}')
print(f'MMSE score at Week 52: Actual: {actual_mmse_scores[i]}\
Predicted: {predicted_mmse_scores[i]}')
print(f'Percent MMSE Score Error at Week 52:\
{round(float(percent_mmse_score_errors[i]),2)}')
pct_sum = sum(percent_slope_errors)
len_pct_err = len(percent_slope_errors)
pct_sum_mmse = sum(percent_mmse_score_errors)
print(f'\nAverage % Slope Error: {round(float(pct_sum/len_pct_err),3)}%')
print(f'Average % MMSE Score Error at Week 52:\
{round(float(pct_sum_mmse/len_pct_err),3)}%')
The error rates generated by a sample run of the Python script above for four patients are
presented in Figure 5.4. Training employed LOOCV and generated AD prognosis for four randomly selected patients, namely, 002_S_2043, 002_S_4171, 002_S_4219, and 002_S_4237. For this
specific run, the best result was obtained for patient 002_S_4219, for whom the cognitive decline
was predicted by the ANN as the line equation y = −0.021x + 29.58, whereas the actual decline
was given by the line equation y = −0.019x + 29.58 (Figure 5.5(c)). Based on this prediction,
patient 002_S_4237 receives an MMSE score of 28.76 one year after the first MMSE score was
recorded. However, the real data shows that the patient’s score was 28.86. The worst prediction
was for patient 002_S_4171 with predicted and actual decline functions of y = −0.032x + 28.27 and
y = 0.058x + 28.27, respectively, corresponding to MMSE scores of 26.71 and 25.25 one year after
the first exam (Figure 5.5(b)).
5.5 Matlab Script for AD Prognosis Using an ANN
In this section we present a sample Matlab script implementing the ANN given in Figure 5.3. The
structure of the program is kept similar to the Python script in Section 5.4 so that readers can easily
compare the two versions.
In lines 10–18, the inputs and outputs for the ANN are saved into X_int and Y, respectively.
The output values are normalized in line 18. In the loop in lines 30–80, four patients are selected
to be tested as part of the LOOCV method. In lines 43 and 45, the ith data point is taken out of the
training set. In lines 48 and 49, the input and output samples are put into table format. Between
lines 52 and 64, the ANN is defined with the number of hidden layers shown in Figure 5.3 and saved
in an object named layers. In lines 67–69, the options for the ANN training are defined, with the
optimization as gradient descent (i.e. adam), the learning rate as 0.01, the number of epochs as 100,
the batch size as 10 (i.e. the number sample inputs to be used at one time during the training) and
the Shuffle option is set to every-epoch (i.e. the input samples are shuffled at each epoch to
reduce over-fitting in training). In line 73, the ANN is trained:
model = trainNetwork(InputsTrain,Outputstrain,layers,options)
and a prediction for the ith patient is generated in line 80:
prediction_scaled = predict(model,LOOCV_inputArray)
where the LOOCV_inputArray holds the ith patient input (this was set in line 38). In lines 82–87,
the prediction is normalized. In lines 89–93, the error percentage for the prediction is computed.
In the rest of the script, the predicted and actual prognosis lines are plotted.
75
76
5 ANNs for Alzheimer’s Disease Prognosis
Figure 5.4 Sample error rates generated by
the Python script for four patients.
5.5 Matlab Script for AD Prognosis Using an ANN
(a)
(b)
(c)
(d)
Figure 5.5 Results generated by the Python script implementing an ANN with 120-60-15-5-1 architecture
shown in Figure 5.3 for four randomly selected patients: (a) 002_S_2043, (b) 002_S_4171, (c) 002_S_4219
and (d) 002_S_4237.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
% The City College of New York, City Univerrwsity of New York
% Written by Sultana Begum
% August, 2024
% GA for Alzheimer disease Prognosis based gene expressions and
% MMSE scores
clear; clc;
% Load training data
training_X = 'training_data/gene_expressions.csv';
X = readtable(training_X);
% Extract input features
X_in = X(:, contains(X.Properties.VariableNames, 'x_at'));
training_Y = 'training_data/mmse_score_slopes.csv';
Y = readtable(training_Y);
mmse_slopes = Y.("m_coef_mmse_points_week_"); % Access column
Y_scaled = rescale(mmse_slopes, 0, 1); % Min-max scaling
% Initialize variables
percent_errors = [];
predictions = [];
actual_slopes = [];
patient_IDs = [];
77
78
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
5 ANNs for Alzheimer’s Disease Prognosis
% Leave-One-Out Cross-Validation
for i = 1:4
fprintf("\n********************************\n");
fprintf("LEAVING OUT ROW %d \n", i);
inputs = X_in;
outputs = Y_scaled;
% Save the input for test sample
LOOCV_input = inputs(i,:);
% Get the slope of test sample
actual_slope = mmse_slopes(i);
% Drop the row with test sample inputs
inputs(i,:) = [];
% Remove the output of test sample
outputs(i) = [];
% Convert data to table format for training
inputsTrain = table2array(inputs);
outputsTrain = table2array(table(outputs, 'VariableNames',{'Output'}));
% Define ANN model: 160-60-15-5-1 architecture
layers = [
featureInputLayer(size(inputsTrain, 2), 'Name', 'input')
fullyConnectedLayer(120, 'Name', 'fc1')
reluLayer('Name', 'relu1')
fullyConnectedLayer(60, 'Name', 'fc2')
reluLayer('Name', 'relu2')
fullyConnectedLayer(15, 'Name', 'fc3')
reluLayer('Name', 'relu3')
fullyConnectedLayer(5, 'Name', 'fc4')
reluLayer('Name', 'relu4')
fullyConnectedLayer(1, 'Name', 'output')
regressionLayer('Name', 'regression')
];
% Set options for training
options = trainingOptions('adam', 'InitialLearnRate', 0.01, ...
'MaxEpochs', 1000, 'Verbose', true, 'MiniBatchSize', 10, ...
'Shuffle', 'every-epoch', 'Plots', 'training-progress');
% Train the model
fprintf("Training Started\n");
model = trainNetwork(inputsTrain, outputsTrain, layers, options);
fprintf("Training Finished\n");
% Prepare LOOCV_input for prediction
LOOCV_inputArray = table2array(LOOCV_input);
% Predict the test sample
prediction_scaled = predict(model, LOOCV_inputArray);
% Normalize prediction
mmse_score_min = min(mmse_slopes);
mmse_score_max = max(mmse_slopes);
mmse_max_min_difference = mmse_score_max - mmse_score_min;
prediction = mmse_score_min + ...
prediction_scaled * mmse_max_min_difference;
% Calculate percent error
rel_err = abs(((prediction - actual_slope) / actual_slope) * 100);
rel_error = round(rel_err, 2);
fprintf("\nThe relative error is: %.2f%%\n", rel_error);
percent_errors = [percent_errors; rel_error]; % Append percent error
% Store predictions and actual values
predictions = [predictions; prediction];
actual_slopes = [actual_slopes; actual_slope];
% Get patient ID and ensure it is a string
patient_ID = X.('patient_id')(i); % Access column
5.5 Matlab Script for AD Prognosis Using an ANN
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
if iscell(patient_ID)
patient_ID = patient_ID{1}; % Convert cell to string
end
patient_IDs = [patient_IDs; patient_ID];
% Plot the results
baseline = Y.("m_baseline_mmse_points_")(i);
x = linspace(0, 52, 7); % 7 evenly spaced values from 0 to 52
% Actual vs Predicted Equation
actual_equation = sprintf("Actual: y = %.3fx + %.2f", actual_slope,...
baseline);
prediction_equation = sprintf("Predicted: y = %.3fx + %.2f", ...
prediction, baseline);
% Plot
figure('Color', 'white', 'Position', [100, 100, 800, 600]);
hold on;
plot(x, actual_slope * x + baseline, 'Color', [0.86, 0.44, 0.58], ...
'LineWidth', 2, 'DisplayName', ...
'Real MMSE Score Decline (best fit)');
plot(x, prediction * x + baseline, 'Color', [0.53, 0.81, 0.92], ...
'LineWidth', 2, 'DisplayName', 'Predicted MMSE Score Decline');
xlabel('Weeks Since First Mini-Mental State Exam');
ylabel('MMSE Score');
xlim([0, 52]);
title(sprintf ...
('MMSE Score Decline\n Predicted vs Actual\n Patient: %s',...
strrep(patient_ID,"_","\_")));
legend('show');
hold off;
% Save the plot
saveas(gcf, sprintf('predictions/%s.png', patient_ID));
end
% Summary of Results
fprintf("****************************\n");
fprintf("LOOP DONE \n\n");
patient_IDs1 = X.('patient_id');
if iscell(patient_IDs1)
patient_IDs1 = string(patient_IDs1);
end
% Print results for each patient
for i = 1:length(percent_errors)
fprintf("Patient ID: %s\n", patient_IDs1(i));
fprintf("Actual Slope of Cognitive Decline: %.3f\n", actual_slopes(i));
fprintf("Predicted Slope of Cognitive Decline: %.3f\n",predictions(i));
fprintf("Percent Error: %.2f%% \n\n", percent_errors(i));
end
% Print average percent error
avg_percent_error = mean(percent_errors);
fprintf('Average Percent Error: %.2f%%\n', avg_percent_error);
Figure 5.6 shows the results generated by the Matlab script for the randomly selected four patients
as in Section 5.4, namely 002_S_2043, 002_S_4171, 002_S_4219 and 002_S_4237 – Figures 5.6(a)
to (d), respectively. We observe that the Matlab script performs similarly to the Python; the error
rates are high but may be improved with better ANN architectures and/or more carefully selected
ANN parameters.
79
80
5 ANNs for Alzheimer’s Disease Prognosis
(a)
(b)
(c)
(d)
Figure 5.6 Sample results generated by the Matlab script implementing an ANN with 120-60-15-5-1
architecture shown in Figure 5.3 for four randomly selected patients: (a) 002_S_2043, (b) 002_S_4171, (c)
002_S_4219 and (d) 002_S_4237.
5.6 Concluding Remarks
With recent advances in machine learning and increased power of personal medical data collection capabilities, it is possible to develop predictive models of neurodegenerative disease prognosis
by mining multimodal datasets. Aggregating information from the ADNI database, where a wide
range of information for AD patients is stored, possible correlations can be explored for cognitive
decline rate of MCI patients and certain continuous personalized measurements. In this chapter,
we show that it is feasible for an ANN to exploit the possible relationship between gene expressions
of AD patients and their cognitive decline measured by MMSE scores taken over time.
Subject matter experts (in this case, biologists and medical researchers) combine their experience with computational experts (e.g. statisticians, software engineers and data scientists) to design
effective ANN architectures (e.g. selection of ANN inputs and outputs, number of hidden layers,
neurons per layer, activation functions, number of epochs and learning rates). Equally important
decisions include selection of biomarkers which may be important, such as blood elements, proteins, MRI information and hippocampal images of the brain [60].
Through the simplistic example presented in this chapter, we show that the use of an ANN in
estimating personalized disease progression for AD patients is worth further research and experimentation. It is our hope that, by integrating effective computational methods and comprehensive
5.7 Exercises for Chapter 5
data sets, a growing body of research focused on enhancing our understanding of early detection
and prognosis of AD will ultimately pave the way for more effective interventions and treatments
in the near future. It is also possible to develop similar tools for other neurodegenerative diseases
such as Parkinson’s disease and amyotrophic lateral sclerosis (ALS), as well as progressive diseases
including various forms of cancer [13].
5.7 Exercises for Chapter 5
1) How can you modify the ANN architecture given in Figure 5.3 so that the prediction power of
the ANN improves?
2) If you reduce the number of hidden layers in the ANN architecture given in Figure 5.3, how will
the performance be impacted? Try different ANN architectures to justify your answer.
3) If you increase the number of hidden layers in the ANN architecture given in Figure 5.3, how
will the performance be impacted? Try different ANN architectures to justify your answer.
4) If you decrease the number of neurons in hidden layers of the ANN architecture given in
Figure 5.3, how will the performance be impacted? Try different ANN architectures to justify
your answer.
5) If you increase the number of neurons in hidden layers of the ANN architecture given in
Figure 5.3, how will the performance be impacted? Try different ANN architectures to justify
your answer.
6) If you increase the number of epochs when training the ANN architecture given in Figure 5.3,
how will the performance be impacted? Try different ANN architectures to justify your
answer.
7) If you decrease the number of epochs when training the ANN architecture given in Figure 5.3,
how will the performance be impacted? Try different ANN architectures to justify your answer.
81
83
6
ANNs for Natural Language Processing
6.1 Introduction
As one of the fastest growing applications of AI, natural language processing (NLP)
aims to allow computers to understand, interpret, and manipulate human language. In this chapter
we present an NLP application that attempts to identify a possible and typically previously unrevealed relationship between words presented in a content (e.g. speech, written document or text
message) and external events happening in various seemingly unrelated venues. Examples of this
type of application include a speech given by a politician followed by rising oil prices, or a famous
singer praising a specific brand of shoes in her concert causing an increased demand on foreign
labour in the shoe industry. NLP is often seen as an intersection between linguistics, computational
disciplines and engineering and involves algorithmic analysis of natural languages [63].
Inputs to NLP algorithms are typically a set of features that are generated based on information collected from external sources or various existing platforms. Information to be explored by an
NLP application could be verbal or written content, from which the input features could be derived
using various approaches (e.g. a set of key words in specific content such as text messages from a
prominent person in a given field). For example, a given set of interaction fragments (e.g. a chain of
emails, voice messages or text message exchanges) can be analysed by an NLP algorithm to explore
whether a connection between them can be formed (e.g. if they discuss the same subject). Similarly,
NLP techniques can identify what topics are covered within a single body of text by defining keywords that are strongly associated with a specific topic or by assigning relevant meanings to words
that have multiple interpretations. In another application, NLP algorithms may explore whether
there is a relationship between a basketball team manager’s pre-game comments and the game
outcome. It is possible that the manager inadvertently uses a weaker form of speech if they do not
have confidence in their team for that game, but cannot say so publicly (e.g. saying "we should
win" versus "we shall win"). If, through NLP, opponents identify such an association, they
may use it against this team in betting sites or by fielding a weaker team and resting strong players for other games. Similarly, political leaders may influence the outcomes of various seemingly
unrelated events through their posts in social media or press conferences they give for news outlets. One can imagine that a fiery speech given by a leader may have a positive or unfavourable
effect on external elements such as financial markets, labour unions or economic policy decisions
(Figure 6.1). These dependencies are often very difficult, if not impossible, to identify without the
help of specialized tools such as NLP.
In general, NLP techniques can be classified into two groups: rule-based and statistics-based
approaches [64]. Decision trees [65] are typical examples for rule-based NLP techniques,
where tree vertices and edges represent conditions and possible outcomes, respectively. Decision
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
84
6 ANNs for Natural Language Processing
Figure 6.1 NLP algorithms can explore possible associations between various forms of natural language
content (e.g. speeches, texts, conversations, sound bites or articles) and other seemingly unrelated events in
different venues (e.g. stock market movements, increasing popularity of soft drinks or labour union
decisions).
trees are useful for pattern matching and parsing content. Statistics-based approaches, on the other
hand, make probabilistic decisions at each decision point in order to assign weights to each possible
outcome. Statistical NLP techniques can express a relative certainty of many different outputs and,
hence, can produce reliable results in complex situations for which unique outcomes may not be
determined.
Question answering (QA) systems are often used in chatbots (i.e. AI-based chatting systems such as Siri) that search for answers in a database or on the internet. QA systems may implement NLP algorithms to first extract information from a question and then search a body of text for
an appropriate answer [66]. In Chapter 11, a chatbot is implemented as a recurrent neural network
(RNN). Text summarization is another application of NLP, where it can identify topics or keywords
used in a body of text to generate sentences that best summarize the text [67].
NLP techniques can also be instrumental for machine translation of a given text segment from
one natural language to another, while keeping the meaning intact. Often during translating
between languages, sentence structure varies according to the syntactical rules of the two languages. NLP algorithms, utilizing ANNs, can be implemented to search for connections between
words and sentence structures in both languages. In Chapter 13, we present a recurrent neural
network with attention mechanism for a simplistic German to English translator.
6.2 Impact of Text Messages on Stock Markets
In this chapter, we present a simple example application of NLP, where an ANN is designed to
explore a possible relationship between text messages sent by an individual and movements of
stock market assets. Specifically, the tweets from former US President Mr. Trump are analysed for
their possible impact on financial securities listed in the S&P 500 index (Standard and Poor’s 500
Index) [68], where the market values of 500 leading publicly traded companies in the USA are listed.
ANNs are a natural fit for implementation of NLP algorithms for this task.
When implementing NLP algorithms for cases similar to this example, proper training of an
effective ANN requires a large set of samples to be used as ANN inputs and corresponding outputs.
For this example, inputs are in the form of content generated in a spoken language (i.e. English)
presented as tweets sent by a person (i.e. former US President Mr. Trump). For each input tweet, the
6.3 A Simple ANN for NLP
corresponding output will be the price movement reported on financial websites after this tweet is
sent (i.e. stocks listed in S&P 500 index).
Often, to reduce noise in input data, natural language content needs to be pre-processed before
being considered as an ANN input. Depending on the application, punctuation marks, words
unrelated to a given subject and some other information regarded as irrelevant may be removed
from content without affecting the presumed relationship between inputs and outputs. The
process of successful trimming of noise from sample data is not a trivial task and depends heavily
on specialized field expertise. For some applications, some words used in a sentence may be
important to determine how self-centered the communication is (e.g. how many times someone
uses the pronoun "you") and others negligible (e.g. usage of the article "the"), whereas in
another application the opposite may true (e.g. in identifying specificity of language used by people
by frequency of using the article "the" but not caring about the number of times a pronoun such
as "you" appears).
6.3 A Simple ANN for NLP
Let us now present a simple ANN to implement an NLP application, where movements of financial
markets are predicted based on the tweets sent by a US president and historical financial data. As
discussed in Chapter 2, in order to construct and operate an effective ANN for a given application,
after selection of ANN inputs and outputs, one has to define the ANN architecture, including the
number of inputs, hidden layers and neurons at each one, selection of activation functions, the
number of epochs and the learning rates to be used during training.
6.3.1 ANN Inputs and Outputs
Although ANN inputs will be taken from the tweets, extracting information from a tweet [69] –
defined as a text message of up to 140 characters – so that it can be employed as an ANN input
is not straightforward. Let us consider defining a set of keywords that may have influence on
financial market movements. For example, it is possible that words like prosper, enrichment
or success may be interpreted by the public as words associated with a successful economy and,
hence, may suggest confidence in markets and influence an increase in stock values. Similarly,
tweets using words like sacrifice, difficulty or failure may hint at difficult times
and may cause markets to fall. However, if this relationship between a set of keywords and stock
market movements were trivial, then everyone would be able to make such inferences and easily
predict outcomes. For realistic cases, there is a lot more subtlety in this relationship, which is why
the selection of keywords requires subject matter expertise.
If a set of keywords is to be used as ANN inputs, then the tweets will be combed to identify
which keywords are used and how often in each tweet. In this way, the number of ANN inputs is
the number of keywords, and the value of each input is the frequency of usage of that keyword in
a given tweet. To form the sample output values for the ANN, historical financial information for
the S&P 500 is used for the days corresponding to the presidential tweets in the input dataset. For a
tweet issued on a particular date, an increase or decrease in the market value is computed for that
day and used as the output value for that tweet.
6.3.2 Keywords
Suppose a set of keywords that may have an influence on financial market movement are
identified with the help of experts from a variety of areas, including sociologists, economists,
85
86
6 ANNs for Natural Language Processing
journalists, historians and scientists. For the example presented here, the 14 keywords selected
are banks, jobs, Xi, stock market, tariff, money, employment, China, dollar,
growth, trade, deal, economy and euro, as shown in Figure 6.2. These keywords are
collected from tweets sent over a period of 13 months. Out of the 7 000 posted tweets, there are
approximately 193 tweets that include one or more of the identified keywords and were sent on
a day that the relevant stock market was operational. Figure 6.3 shows a sample set of tweets,
where the keywords are highlighted. The number of keywords is the number of inputs of the ANN.
For a given tweet, each input value reflects how many times that keyword is used in that tweet.
Figure 6.4 shows the ANN inputs based on the tweets in Figure 6.3.
In addition to the frequency of each keyword used in a tweet, historical tick data is also employed
as part of the ANN input. In this example, the opening price of a day and the volume traded on that
day (an example is shown in Figure 6.5) are included in the ANN inputs. Percentage change of daily
price is the ANN output.
6.3.3 Formation of Training Data
Tweets are first pre-processed by removing images, hyperlinks and re-tweet signatures. Also, tweets
from weekends and holidays are shifted to the next day that the market was open. There are 193
tweets suitable for training (i.e. those that contain one or more keywords), and, hence, the ANN
Figure 6.2
example.
Figure 6.3
Keywords selected for the NLP
Subset of tweets used in forming the ANN inputs for the NLP example.
6.3 A Simple ANN for NLP
Figure 6.4 Sample tweets given in Figure 6.3 are processed so that the number of times each keyword is
used in each tweet constitutes part of the ANN input.
Figure 6.5
Sample historical tick data used in forming ANN inputs and outputs.
input matrix has dimensions of (193 × 16), since there are 14 keywords and two tick data entries
(i.e. opening price and volume). The ANN output matrix dimensions are (193 × 1) since there is
one ANN output per sample.
Two example inputs for two tweets are shown in Figure 6.6, where first 14 elements are the
frequency of each keyword used in each tweet followed by the opening price and the volume traded
for that day. In the first tweet, the third and eighth keywords are used once, whereas in the second
one the fourth, fifth and tenth keywords are used (the fourth keyword is used twice and the others
once).
For this example, the output of the ANN is defined as the percentage change in asset price. To
define each output, the closing price of a given day compared to the price of the following day is
computed and assigned as the output for the corresponding inputs. However, since their values
are very large compared to the keyword frequencies, it is necessary to normalize closing price
and traded volume values between 0 and 1; otherwise, keywords will be treated like noise due to
87
88
6 ANNs for Natural Language Processing
Figure 6.6 Two input examples for an ANN implementing the NLP example: the first 14 elements are the
frequency of each keyword used in each tweet; these are followed by the opening price and the volume
traded for that day.
their comparatively small magnitudes of input values and, hence, will have little impact on output
computations. This information is readily available from historical tick data as shown in Figure 6.5.
6.3.4 ANN Architecture
A fully-connected ANN architecture is selected for this NLP example as shown in Figure 6.7, where
there are 16 inputs (14 keywords, opening price and traded volume), two hidden layers with four
and eight neurons, respectively, and one output layer (percentage change in closing price). As noted
in earlier chapters, inputs, the number of inputs, hidden layers, neurons in each hidden layer, outputs and other ANN architecture parameters are selected based on collaboration among various
subject matter experts and software engineers.
The training process for the ANN defined in this example can be summarized as follows. Initially,
random values are assigned to the weights and biases of the ANN. Then, the 193 input samples are
propagated through the ANN by forming an input matrix with size of (193 × 16) as described in
Chapter 2. For each input sample, an error is calculated based on the difference between the generated ANN output and the historical financial market data for that day. Using back propagation,
the error values for weights and biases are reduced by updating the weight and bias values. After
a pre-determined number of epochs, the training process ends, at which point the ANN is ready
to generate predictions of financial market movements based on new tweets. When there is a new
Figure 6.7 Fully-connected ANN architecture to be used for NLP implementation: 16 inputs (14 keywords,
opening price and traded volume), two hidden layers with 8 and 4 neurons, respectively, and one output.
6.4 Python Script for NLP Using an ANN
tweet, it is processed to determine how many times each of the 14 keywords appears in it. The
resulting 14 inputs, together with the opening price and traded volume, are then fed into the ANN
to generate an output, which will be the prediction of finance market changes for the next day.
6.4 Python Script for NLP Using an ANN
A sample Python script implementing the ANN described in this chapter is presented below.
Functions to clean undesirable characters from tweets, print ANN weights, normalize input
columns, define keywords, eliminate invalid days and prepare ANN inputs and outputs are presented in the first section of the code. In line 54, the keywords are searched for in each tweet and
stored in matrix called df. In lines 118 and 119, sample inputs and outputs are saved in arrays
named x and y, respectively. An ANN object named neural_net is created in line 121. The ANN
input layer is defined in line 125. Two hidden layers with four and eight neurons are created in
lines 129 and 132, respectively. The output layer with a linear activation function (i.e. it outputs the
values as is without filtering negative values, etc.) is declared in line 135. In line 137, optimization
parameters are defined as gradient descent algorithms (i.e. option adam) using mean absolute error
computation. In line 141, the ANN is trained as follows:
neural_net.fit(x, y, epochs = 1200)
The weights of the trained ANN are printed in lines 143 and 144. In the loop starting in line
151 and ending at line 166, the trained ANN is run for the number of days in the variable
days_to_predict, and the number of successful predictions and the error are recorded (lines
164–166). Finally, the results are stored in files in lines 171 to 180.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# The City College of New York, City University of New York
# Written by Kelvin Ma
# August, 2020
# NLP: Predicting financial stock movement based on tweets from Pres. Trump
import re
import datetime
from datetime import date
import math
import numpy as np
import pandas as pd
import tensorflow as tf
# Function to remove punctuation and web links from tweets
# @\S+|https?://\S+ - matches either a substring which starts with @
# and contains non-whitespace characters \S+ OR a link(url) which
# starts with http(s)://
def clean_tweet(tweet):
return ' '.join(re.sub('(@[A-Za-z0-9]+)|([ ̂ 0-9A-Za-z \t])|(\w+:\/\/\S+)',
' ', tweet).split())
def str_to_date(string):
return datetime.datetime.strptime(string, '%Y-%m-%d').date()
def print_weights(weights):
# weights = model.get_weights();
print('\n******* WEIGHTS OF ANN *******\n')
for i in range(int(len(weights)/2)):
print('Weights W%d:\n' %(i), weights[i*2])
print('Bias b%d:\n' %(i), weights[(i*2)+1])
def normalize_column(dataframe, col_name):
maximum = max(dataframe[col_name])
minimum = min(dataframe[col_name])
89
90
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
6 ANNs for Natural Language Processing
if maximum != minimum:
dataframe[col_name] = (dataframe[col_name] - minimum)/(maximum-minimum)
return dataframe[col_name]
# Load in file, remove punctuation and weblinks from all tweets and then
#
perform sentiment analysis.
print('\n\n********** CLEANING TWEETS **********\n\n')
filename = "trumptwitter_2019_01_01_to_2020_03_11.csv"
df = pd.read_csv(filename)
df["Clean Tweet"] = df['text'].apply(lambda x: clean_tweet(x))
df['created_at'] = pd.to_datetime(df['created_at']).dt.date
# Data cleaning and processing
# Check if the tweet contains any keywords - remove all tweets that don't
print('\n\n********** IDENTIFYING KEY WORDS **********\n\n')
key_words = ['Europe', 'China', 'tariff', 'Stock Market', 'economy', 'bank',
'trade', 'jobs', 'money', 'dollar','currency','Xi','deal','growth']
df['noof_keywords'] = np.where(df.text.str.contains('|'.join(key_words)),1,0)
for key_word in key_words:
df[key_word] = np.where(df.text.str.contains(key_word), 1, 0)
# list of invalid days: including federal holidays, days with no data
invalid_days = ['2019-01-01', '2019-01-02', '2019-01-21', '2019-02-18',
'2019-04-19', '2019-05-27', '2020-05-08', '2019-07-04',
'2019-09-02', '2019-11-28', '2019-12-25', '2020-01-01',
'2020-01-20','2020-02-17', '2020-04-10',]
missing_days = ['2019-01-21', '2019-02-18', '2019-05-27', '2019-09-02',
'2020-01-20','2020-02-17']
noof_missing_days = len(missing_days)
invalid_days = list(map(lambda x:str_to_date(x),invalid_days))
# if the tweets are in invalid days, move them to the next day for prediction.
for index, row in df.iterrows():
if row['created_at'] in invalid_days:
new_day = row['created_at'] + datetime.timedelta(days=1)
df.loc[index, "created_at"] = new_day
# Move tweets sent on a saturday or sunday to the nearest Monday.
# When using the weekday() function -> 0 is Monday and 6 is Sunday
for index, row in df.iterrows():
if row['created_at'].weekday() == 5:
new_day = row['created_at'] + datetime.timedelta(days=2)
df.loc[index, "created_at"] = new_day
elif row['created_at'].weekday() == 6:
new_day = row['created_at'] + datetime.timedelta(days=1)
df.loc[index, "created_at"] = new_day
# group the data by date - aggregate all keywords used and sum them up.
# for example, the tweets from the day mention 'dollar' multiple times, add
#
them together.
columns_to_aggregate = {'noof_keywords':'sum'}
for key_word in key_words:
columns_to_aggregate[key_word] = 'sum'
grouped_df=df.groupby(by='created_at').agg(columns_to_aggregate).reset_index()
#% Stock prices
filename = "S&P500_2019_01_01_to_2020_03_11.csv"
snp_df = pd.read_csv(filename)
# return at close each day
snp_df['percent change'] = snp_df['Close'].pct_change()
missing_days = list(map(lambda x:str_to_date(x), missing_days))
for missing_day in missing_days:
grouped_df = grouped_df[grouped_df['created_at'] != missing_day]
# ANN
# create the input data: last 100 days will be used for predictions at the end
# this window is subject to change
grouped_df['day_opening'] = snp_df['Open']
grouped_df['units_traded'] = snp_df['Volume']
days_to_predict = 100+noof_missing_days
6.4 Python Script for NLP Using an ANN
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
# before putting any data into NN, we normalize the data (between 0 and 1)
# Normailize only the open and volume traded (beacuse they are large compared
# to the other input values)
grouped_df['day_opening'] = normalize_column(grouped_df, 'day_opening')
grouped_df['units_traded'] = normalize_column(grouped_df,'units_traded')
features = ['noof_keywords', 'day_opening', 'units_traded']
features = features + key_words
num_inputs = len(features)
x = np.array(grouped_df[1:(-1*days_to_predict)][features])
y = np.array(snp_df[2:((-1*days_to_predict)+1)][['percent change']])
neural_net = tf.keras.Sequential()
# NN inputs: how many times each keyword was mentioned by President,
# opening price of the stock and volume traded
neural_net.add(tf.keras.layers.Dense(num_inputs, activation='relu',
input_shape=(num_inputs,)))
## add another hidden layer with 4 neurons to the NN
neural_net.add(tf.keras.layers.Dense(4, activation='relu'))
## add another hidden layer with 8 neurons to the NN
neural_net.add(tf.keras.layers.Dense(8, activation='relu'))
## add an output layer with a single output (percent change)
neural_net.add(tf.keras.layers.Dense(1, activation='linear'))
neural_net.compile(optimizer='adam', loss='mean_absolute_error')
## train the ANN model using 1200 iterations
print('\n\n********** Begin ANN training **********\n\n')
neural_net.fit(x, y, epochs=1200)
weights = neural_net.get_weights()
print_weights(weights)
print('\n\n********** ANN training complete **********\n\n')
# make predictions
noof_correct_movement = 0
noof_predictions = 0
diffs = []
input_features = []
print('\n\n********** ANN PREDICTIONS **********\n\n')
for i in range(days_to_predict, noof_missing_days, -1):
input_features.clear()
noof_predictions += 1
actual_change = snp_df.iloc[(-1*i)+1]['percent change']
for feature in features:
input_features.append(grouped_df.iloc[-1*i][feature])
predicted_change = neural_net.predict(np.array([input_features]))[0,0]
print(f"The predicted change for{grouped_df.iloc[(-1*i)+1]['created_at']}",
end='')
print(f" was: {predicted_change:.5f}")
print(f"Actual change was for {snp_df.iloc[(-1*i)+1]['Date']} was: ",
end='')
print(f"{round(actual_change,4)}\n")
if (predicted_change * actual_change) > 0:
diffs.append(math.fabs(predicted_change - actual_change))
noof_correct_movement += 1
percent_correct = (noof_correct_movement/noof_predictions) * 100
print(f"ANN was correct in predicting the movement ", end = '')
print(f"{((noof_correct_movement/noof_predictions) *100):.2f}% of the time in",
end='')
print(f" {noof_predictions} predictions.")
average_diff = round(sum(diffs)/len(diffs),5)
print(f"The average error of the correct predictions were", end='')
print(f" {average_diff* 100.0:.1f} %")
with open('output.txt','a') as output_file:
output_file.write(f'{date.today()} percentage of times correct prediction')
output_file.write(f' {round(percent_correct,1)} % ')
output_file.write(f' with error {average_diff* 100.0:.1f}%\n')
grouped_df.to_csv('TrumpTweet_Input.csv')
91
92
6 ANNs for Natural Language Processing
Figure 6.8 Output generated by a sample run of the Python script: out of 100 runs, 62 runs ANN made a
correct prediction, where the average error of correct predictions is 0.5%.
A sample run of the Python script given above is shown in Figure 6.8, where out of 100 runs
of the ANN, 62% of the time the correct prediction is obtained where the average error of correct
predictions is 0.5%.
6.5 Matlab Script for NLP Using an ANN
A sample Matlab script implementing the ANN in Figure 6.7 is given below. For ease of comparison,
the Matlab implementation is kept similar to the Python script given in Section 6.4.
In lines 13 and 16, tweets and stock prices are read into arrays called df and snp_df, respectively.
Between lines 19 and 111, tweets and tick data are manipulated so that they will have no empty
days, weekends or holidays, or any other invalid data. Sample inputs and outputs are saved in arrays
named X and y in lines 113 and 114, respectively. Between lines 119 and 132, an input layer with
16 neurons, hidden layers with four and eight neurons and an output layer are created and saved
in an object named layers. After defining the options in lines 134 and 135, the ANN is trained in
line 136 as follows:
net = trainNetwork(X, y, layers, options);
In the loop given between lines 157 and 173, different inputs are tried and the results are recorded
as to whether the outputs have a correct increase or decrease prediction.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
% The City College of New York, City University of New York
% Written by Olga Chsherbakova
% December, 2023
% NLP: Predicting financial stock movement based on tweets from Pres. Trump
clc;
clear;
% Function to remove punctuation and web links from tweets
% @\S+|https?://\S+ - matches either a substring which starts with @
% and contains non-whitespace characters \S+ OR a link(url) which
% starts with http(s)://
% Read files
df = readtable('trumptwitter_2019_01_01_to_2020_03_11.csv', ...
'VariableNamingRule', 'preserve');
% Stock prices
snp_df = readtable('S&P500_2019_01_01_to_2020_03_11.csv', ...
'VariableNamingRule', 'preserve');
6.5 Matlab Script for NLP Using an ANN
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
% Load in file, remove punctuation and weblinks from all tweets and then
%
perform sentiment analysis.
disp('********** CLEANING TWEETS **********');
df.CleanTweet = cellfun(@clean_tweet, df.text, 'UniformOutput', false);
df.created_at = datetime(df.created_at, 'InputFormat', 'yyyy-MM-dd HH:mm');
df.created_at = dateshift(df.created_at, 'start', 'day');
% Data cleaning and processing
% Check if the tweet contains any keywords - remove all tweets that do not
disp('********** IDENTIFYING KEY WORDS **********');
keyWords = {'Europe', 'China', 'tariff', 'Stock Market', 'economy', ...
'bank', 'trade', 'jobs', 'money', 'dollar', 'currency', ...
'Xi', 'deal', 'growth'};
for i = 1:length(keyWords)
df.(keyWords{i}) = contains(df.text, keyWords{i});
end
df.noof_keywords = sum(contains(df.text, keyWords), 2);
% Processing invalid and missing days
snp_df.Date = datetime(snp_df.Date, 'InputFormat', 'yyyy-MM-dd');
% list of invalid days: including federal holidays, days for when
% ther is no data
invalidDays = {'2019-01-01', '2019-01-02', '2019-01-21', '2019-02-18', ...
'2019-04-19', '2019-05-27', '2020-05-08', '2019-07-04', ...
'2019-09-02', '2019-11-28', '2019-12-25', '2020-01-01', ...
'2020-01-20', '2020-02-17', '2020-04-10'};
missingDays = {'2019-01-21', '2019-02-18', '2019-05-27', '2019-09-02', ...
'2020-01-20', '2020-02-17'};
noof_missing_days = length(missingDays);
invalidDays = datetime(invalidDays, 'InputFormat', 'yyyy-MM-dd');
missingDays = datetime(missingDays, 'InputFormat', 'yyyy-MM-dd');
for i = 1:height(df)
if ismember(df.created_at(i), invalidDays)
df.created_at(i) = df.created_at(i) + days(1);
end
if weekday(df.created_at(i)) == 1 % Sunday
df.created_at(i) = df.created_at(i) + days(1);
elseif weekday(df.created_at(i)) == 7 % Saturday
df.created_at(i) = df.created_at(i) + days(2);
end
end
% Group Data
grouped_df = group_data(df, keyWords);
% We exclude weekends and holidays
grouped_df = grouped_df(∼isweekend(grouped_df.created_at) & ...
∼ismember(grouped_df.created_at, invalidDays),:);
% Convert created_at to datetime for synchronization
grouped_df.created_at = datetime(grouped_df.created_at, ...
'InputFormat', 'yyyy-MM-dd');
% Filter data based on missing days
for i = 1:length(missingDays)
grouped_df = grouped_df(grouped_df.created_at ∼= missingDays(i),:);
end
% ANN
% create the input data - the last 100 days will be used for predictions
% at the end
% this window is subject to change
snp_df.percent_change = [NaN; diff(snp_df.Close)./ snp_df.Close(1:end-1)];
snp_temp = table(snp_df.Date, snp_df.Open, snp_df.Volume, ...
snp_df.percent_change,'VariableNames', {'Date', 'Open', ...
'Volume','PercentChange'});
% Data merging
synchronized_data = innerjoin(grouped_df, snp_temp, 'LeftKeys', ...
'created_at', 'RightKeys', 'Date');
93
94
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
6 ANNs for Natural Language Processing
% before putting any data into NN, we normalize the data (between 0 and 1)
synchronized_data.day_opening = synchronized_data.Open;
synchronized_data.units_traded = synchronized_data.Volume;
% Normailize only the open and volume traded (beacuse they have large
% values compared to the other input values)
synchronized_data.day_opening = normalize_column(synchronized_data,...
'day_opening');
synchronized_data.units_traded = normalize_column(synchronized_data,...
'units_traded');
% Preparing data for a neural network
features = ['noof_keywords', keyWords, 'day_opening', 'units_traded'];
num_inputs = length(features);
% Make sure there are no NaN values in the data
if any(any(isnan(synchronized_data{:, features}))) || ...
any(isnan(synchronized_data.PercentChange))
error('Invalid or missing data contains NaN values.Cleaning required.');
end
X = table2array(synchronized_data(:, features));
y = table2array(synchronized_data(:, 'PercentChange'));
% Creation and training of a neural network
% NN inputs: how many times each keyword was mentioned by President,
% opening price of the stock and volume traded
layers = [
featureInputLayer(num_inputs)
fullyConnectedLayer(50)
reluLayer
% add another hidden layer with 4 neurons to the NN
fullyConnectedLayer(4)
reluLayer
% add another hidden layer with 8 neurons to the NN
fullyConnectedLayer(8)
reluLayer
% add an output layer with a single output (percent change)
fullyConnectedLayer(1)
regressionLayer
];
options = trainingOptions('adam', 'MaxEpochs', 42, 'MiniBatchSize', 10,...
'InitialLearnRate', 1e-4, 'Verbose', 0, 'Plots', 'training-progress');
net = trainNetwork(X, y, layers, options);
% Print weights
print_weights(net);
% make predictions
noof_correct_movement = 0;
noof_predictions = 0;
diffs = [];
% Adjust based on available data
days_to_predict = min(height(synchronized_data), 100);
start_index = max(1, height(synchronized_data) - days_to_predict + 1);
if height(synchronized_data) < days_to_predict
error('Not enough data to make predictions.');
end
disp('********** ANN PREDICTIONS **********');
for i = start_index:height(synchronized_data)
input_features = table2array(synchronized_data(i, features));
actual_change = synchronized_data.PercentChange(i);
predicted_change = predict(net, input_features);
fprintf('\nThe predicted change for %s was: %.5f\n', ...
string(synchronized_data.created_at(i)), predicted_change);
6.5 Matlab Script for NLP Using an ANN
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
fprintf('Actual change was for %s was: %.4f\n', ...
string(snp_df.Date(i+1)), actual_change);
if (predicted_change * actual_change) > 0
diffs(end+1) = abs(predicted_change - actual_change);
noof_correct_movement = noof_correct_movement + 1;
end
noof_predictions = noof_predictions + 1;
end
if noof_predictions == 0
disp('No predictions were made.');
else
percent_correct = (noof_correct_movement / noof_predictions) * 100;
average_diff = mean(diffs);
fprintf('\nANN was correct in predicting the movement ');
fprintf('%.2f%% of the time in %d predictions.\n', percent_correct,...
noof_predictions);
fprintf('The average error of the correct predictions was '); ...
fprintf('%.1f%%\n', average_diff * 100);
end
% Saving results
writetable(df, 'TrumpTweet_Input.csv');
% Functions
function cleanText = clean_tweet(tweet)
cleanText = regexprep(tweet, ...
'(@[A-Za-z0-9]+)|([ ̂ 0-9A-Za-z \t])|(\w+:\/\/\S+)', ' ');
end
function normalizedColumn = normalize_column(data, columnName)
maxVal = max(data.(columnName));
minVal = min(data.(columnName));
normalizedColumn = (data.(columnName) - minVal) / (maxVal - minVal);
end
function grouped = group_data(df, keyWords)
% Group data by date
varsToSum = ['noof_keywords', keyWords];
grouped = varfun(@sum, df, 'GroupingVariables', 'created_at',...
'InputVariables', varsToSum);
grouped.Properties.VariableNames(end-length(varsToSum)+1:end) = ...
varsToSum;
end
function isWeekend = isweekend(date)
dayOfWeek = weekday(date);
% 1 is Sunday, 7 is Saturday
isWeekend = (dayOfWeek == 1) | (dayOfWeek == 7);
end
function print_weights(net)
disp('******* WEIGHTS OF ANN *******');
for i = 1:length(net.Layers)
layer = net.Layers(i);
if isprop(layer, 'Weights')
fprintf('\nWeights W%d:\n', i);
disp(layer.Weights);
end
if isprop(layer, 'Bias')
fprintf('Bias b%d:\n', i);
disp(layer.Bias);
end
end
end
In Figure 6.9 the output generated by a sample run of the Matlab script is shown, where, out of
100 runs of the ANN, 62 correct predictions are obtained with an average error of 0.9%.
95
96
6 ANNs for Natural Language Processing
Figure 6.9 Output generated by a sample run of the Matlab script: out of 100 runs of the ANN, 62 correct
predictions are obtained with an average error of 0.9%.
6.6 Concluding Remarks
Posts in social media, articles appearing in the printed press and comments made in press
conferences for news outlets may affect the outcomes of various external and seemingly unrelated
events (e.g. financial securities values, company valuations or employment decisions). NLP
techniques allow interactions between natural language content (e.g. spoken, written or posted
text) and external events (e.g. house prices) to be explored. ANNs can be used to implement
statistics-based NLP techniques by first extracting keywords from a body of text and then associating them with external event recordings temporally related to this content. These keywords,
together with their usage frequencies, may be passed into an ANN that can be trained to predict
the outcome of various external events.
In this chapter, tweets from former US President Trump are used to explore their possible impact on financial securities listed on the S&P 500 index. First a set of keywords used
in tweets is identified that could potentially be related to financial asset movements. The
numbers of these keywords tweeted each day together with opening price and daily traded
volume are used as inputs to an ANN. Prediction of stock price movements is the output
of the ANN. Using tweets and historical financial data corresponding to those days that
tweets were published, an ANN is trained to predict future stock movements using future
tweets.
Success in employing ANNs for the NLP class of problems in real-life situations depends on
correct identification of inputs and outputs for an ANN, appropriate selection of keywords and
design of the ANN architecture. The complexity of real-life problems that can be solved by AI
and NLP techniques require the combined experience and skill-sets typically available to various
subject matter experts (e.g. scientists, money managers, business executives, medical personnel
and others) and computing professionals (e.g. statisticians, mathematicians, computer scientists
and software engineers).
6.7 Exercises for Chapter 6
6.7 Exercises for Chapter 6
1) Explore a possible relationship between the tweets issued from Centers for Disease Control
and Prevention (CDC) about COVID-19 pandemic updates and the movement of Delta Airline
stocks. It is expected that if CDC relaxes conditions for isolation, travel stocks will gain value.
Use tweets issued between June 2020 and Aug 2020.
● Use an ANN similar to the one in Figure 6.7.
● Define your own keywords.
● Experiment with the ANN architecture to improve NLP performance.
● Experiment with different sets of keywords to improve NLP performance.
● Suggest ways to improve performance of the NLP.
2) Repeat Exercise 1 for tweets from the Food and Drug Administration (FDA) to predict the
movement of Moderna stocks.
3) Repeat Exercise 1 for Google twitter feed and Google stocks. Use tweets issued between June
2020 and Aug 2020.
4) Repeat Exercise 1 for Apple twitter and Apple stocks.
5) Repeat Exercise 1 for CNN News twitter and the Dow Jones market index.
6) The City University of New York (CUNY) is the largest public higher education system in the
USA with 25 colleges under its umbrella, serving more than 250 000 students. Any decision
of using remote meeting tool Zoom is expected to have a major impact on Zoom stock prices.
Repeat Exercise 1 for tweets from the CUNY chancellor and movement of Zoom stocks.
97
99
7
Convolutional Neural Networks
7.1 Introduction
It is understood that when humans process visual information, they condense a broader visual field
of view into multiple overlapping smaller receptive fields that are controlled by respective neurons
in different areas of the brain (i.e. visual cortices). In order to recognize an object, a visual input is
processed through hierarchically-connected receptive fields. Neurons that control receptive fields
respond to relatively small and simple visual stimuli, whereas neurons associated with receptive
fields located deeper in the brain process increasingly larger and more complex optical features. As
visual information progresses along the hierarchically connected receptive fields, neurons begin to
represent visual objects in progressively more accurate ways until an object can be recognized by
the brain (Figure 7.1a).
Figure 7.1b shows the growth of receptive field sizes as the brain processes visual stimuli by
passing them from the primary visual cortex (called V1) through secondary visual cortices (called
V2 and V4) and, finally, recognizing the object at its inferior temporal (IT) cortex. Figure 7.1c shows
how neurons process an image as the brain processes it through the layers, each with increasingly
larger receptive fields (indicated by the red path in Figure 7.1a).
Convolutional neural networks (CNNs) are inspired by processes occurring in the
brain region that receive, integrate and process visual information transmitted from the retinas of
human eyes. Deep neural networks (DNNs) are a class of ANNs characterized by having two
or more hidden layers. CNNs belong to a class of DNNs whose hidden layers typically consisting
of a series of convolutional layers (CLs), each converting an input image into smaller
sized information via a multiplication or a dot-product operation. A typical CL uses multiple
small feature detectors, called convolutional filters, that are individually slid over a
larger image and, by performing a weighted sum operation, generate an activation map indicating
the presence or absence of a given feature within the image. The number of convolutional filters
used in a given CL is dictated by the number of features that need to be detected by the CNN. For
example, CNNs for line detection typically use a low number of filters (e.g. 10), whereas detection
of complex features may require over 100 filters.
Another type of CNN layer is called a pooling layer (PL), whose role is to down-sample
(i.e. reduce the dimensions of a matrix without losing important information contained by its
elements) activation maps to reduce amount of data to be processed. A PL uses a filter to perform
down-sampling, where a filter of size (f × f ) can merge the pixels (i.e. the smallest image elements)
in adjacent (f × f ) cells into a single cell by replacing them with the maximum value among the
cells. A rectified linear unit (RELU) of a CNN is a non-linear activation function to
control propagation of information between adjacent layers. Typically, a RELU removes negative
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
100
7 Convolutional Neural Networks
(a)
(b)
(c)
Figure 7.1 Feature recognition by the visual hierarchy of brain cortices (adapted from [70] / with
permission of Grace Lindsay).
Figure 7.2 Main components of a CNN for colored images: one convolution, pooling and unrolling layers, a
RELU and a fully-connected ANN.
values from an activation map by setting them to zero. This is analogous to the activation function
used in the neurons of an ANN in Chapter 2. After the last CL of an CNN, activation maps are
“unrolled” by converting their input matrices into a one-dimensional vector and then concatenating them. This layer, called the unrolling layer (UL), generates an input vector, which
is then sent to a fully-connected ANN to generate the outputs of the CNN. Figure 7.2 shows a
simple CNN processing an image by passing it through a CL, a RELU and a PL before unrolling
the results to be fed into a fully-connected ANN to generate outputs.
7.1.1
Training CNNs
The weight coefficients used in CNN filters and ANN links are initially assigned random values.
In a training process similar to that in ANNs described in Chapter 2, values of these weights
are evolved through consecutive forward and back propagation steps, called epochs. A simple
algorithm for training a CNN is shown in Figure 7.3. First the structure of the CNN and the number
of epochs to be used for training are defined. Next, weights and biases for all layers are initialized
to random values. During the forward propagation phase, at each CL, a convolution is performed
over the input matrix with each filter. After convolution with all filters has been completed, RELU
7.3 Applications of CNNs
Figure 7.3
A simple algorithm for training a CNN.
is applied to the CL outputs and passed to the PL. Finally, UL unrolls the outputs of the PL and
feeds it to a fully-connected ANN. With forward and back propagation operations performed by
the ANN, a preliminary output with its associated error are generated. Back propagation of the
ANN is continued through the CNN layers to compute the error and adjust the weights and biases
at the inputs of the CNN. This process is repeated until a pre-determined number of epochs have
been reached. If there are multiple layers of CLs and PLs in a CNN, these operations are repeated
consecutively for each layer.
7.2 Variations of CNNs
The number of layers in CNNs may range from a few to hundreds [71] as their classification power
varies from simple tasks such as recognition of digits to identifying thousands of different classes of
information. For example, a relatively simple CNN called LeNet is designed to recognize printed
and handwritten digits [72]. It processes an input image represented by a (32 × 32) matrix through
three CLs before passing it to a fully-connected ANN that generates 10 outputs (i.e. one for each
recognized digit from 0 to 9).
As another example, a recent CNN designed for classifying an image into one of 1000 given categories, called GoogLeNet, employs a total of 22 layers [73]. It has 2 CLs and 10 so-called inception
modules (i.e. layers) that reduce computational cost and avoid overfitting by using many different
sizes of filters at every inception module [74]. GoogLeNet was trained with 1.2 million images to
perform classification on images with up to 352 pixels along their shorter dimensions.
7.3 Applications of CNNs
The most common application of a CNN is image recognition. For automation of mail and package
sorting, post offices and shipping companies use CNNs to recognize hand-written city and country
101
102
7 Convolutional Neural Networks
codes to determine where to dispatch the letters and packages [75]. CNNs are well-suited to solving
computer vision problems that aim to gain high-level understanding of content from digital images
and videos. Using CNNs for computer vision applications, for example, allowssearch engines
to analyse, understand and categorize digital images in the public domain. Another successful
application of CNNs is categorization of legal documents. Traditionally done by clerical employees,
CNNs can assign documents to appropriate lawyers. A CNN analyses a legal document by exploring the word order in each sentence and predicts its topic [76]. A well-designed and trained CNN
may accurately categorize legal documents as, for example, subpoenas, affidavits or complaints.
7.4 CNN Components
In general, a CNN is a neural network with special convolutional and pooling layers that perform
operations on small regions of their inputs. These layers allow a CNN to handle large volumes
of inputs and learn relationships between adjacent input pixels. CNNs are suited to applications
where the inputs have spatial significance, such as pixels in image recognition or the words of a
natural language sentence. The following are the different layers defined for a CNN (Figure 7.4):
Convolutional layer (CL): scans an input for a given set of features by performing convolutions
on the input image using small filters; it is always followed by a RELU.
● Recti-linear unit (RELU): applies the RELU activation function to its input, which is the
output of a CL, setting all negative values to zero.
● Pooling layer (PL): reduces the dimensions of its input, which is typically from a RELU, by
keeping only the maximum value of local regions in its input.
● Unrolling layer (UL): reshapes the 3-D input, which is typically from a PL or a ReLU, into a
1-D column of values that are readable by an ANN.
● ANN: one or more fully-connected layers to generate CNN outputs.
●
7.5 A Numerical Example of a CNN
Let us now consider an numerical example to classify a simple input image looking either like an
X or an O, as shown in Figure 7.5. These two shapes are typically referred to as the two classes
Figure 7.4
Layers of a simple CNN.
Figure 7.5
Two classes of images that the example CNN is expected to recognize.
7.5 A Numerical Example of a CNN
that the CNN will be designed to recognize. Therefore, there are two outputs for the CNN, namely
Y1 and Y2 , defined as follows:
●
●
If an input image is classified as an O, the outputs are: Y1 = 1, Y2 = 0
If an input image is classified as an X, the outputs are: Y1 = 0, Y2 = 1
Suppose there are two filters, each of size (2 × 2), initialized to random values. For ease of
visualization, we select filter values as 1 and −1. They can also be assigned decimal values as
shown for the biases in Figure 7.6. Note that the size of a CL filter has to be smaller than the input
matrix over which it will operate. In the example below, a filter of (2 × 2) is defined to operate over
a (4 × 4) image. There may be multiple filters defined for a CL. Each filter corresponds to a feature
that image may contain (e.g. the ears or eyes of a cat). If there are multiple CLs, there will be
separate unique filters for each CL. For each CL filter, there is a bias, whose value is also initialized
randomly. One may consider the CL filters as equivalent to weights in an ANN. As CNN training
progresses through forward and backpropagation, the initial random filter shapes are updated to
identify key features of an input image.
Let us now consider the sample input with a size of (4 × 4) shown in Figure 7.7. The goal of the
example CNN is to classify its input as an X or an O. The input image shown in Figure 7.7 should
be classified by the CNN as a circle since it looks more like an O than an X.
Convolution layer with Filter 1: Figures 7.8 and 7.9 illustrate the steps of convolution when
Filter 1 is convolved with the input image given in Figure 7.7. At each step, the numerical contents
Figure 7.6
Two filters defined for the example application of a CNN.
Figure 7.7 Example input image to be identified as either an X or an O by the example CNN (it clearly
looks like an O).
Figure 7.8
Convolutions with Filter 1.
103
104
7 Convolutional Neural Networks
Figure 7.9
Convolutions with Filter 1 (continued).
of Filter 1 are multiplied with the contents of the corresponding image pixels. Since the size of
Filter 1 is (2 × 2) and it is convolved with a (4 × 4) image, there are a total of 9 steps to complete the
convolution with Filter 1, as shown in Figures 7.8 and 7.9.
Convolution layer with Filter 2: Convolution of the input image given in Figure 7.7 with the
elements of Filter 2 are shown in Figures 7.10 and 7.11. Similar to the case of Filter 1, there are 9
steps for this operation since Filter 2 of size (2 × 2) convolves with a (4 × 4) image.
RELU operations: After CL operations are complete, the resulting matrix has smaller dimensions. Reduction in dimensions after CL operations depends on the dimensions of the filters. For
this example, CL generates convolution matrices, called C1 and C2 , whose elements are results of
convolutions performed using Filters 1 and 2, respectively. The dimensions of the input matrix
are (4 × 4), whereas the C1 and C2 matrices have dimensions of (3 × 3) since CL uses two filters
with dimensions of (2 × 2). Elements of C1 correspond to the results of the convolution operations
shown in Figures 7.8 and 7.9. Similarly, the elements of C2 are generated by the convolution operations shown in Figures 7.10 and 7.11. RELU first replaces the negative-valued elements in C1 and
C2 by zero, generating f (C1 ) and f (C2 ), respectively, as shown in Figure 7.12b. It then normalizes
Figure 7.10
Convolutions with Filter 2.
7.5 A Numerical Example of a CNN
Figure 7.11
Convolutions with Filter 2 (continued).
(a)
(b)
(c)
Figure 7.12 RELU operations: (a) form matrices, (b) apply RELU to C1 and C2 and eliminate negative values,
(c) normalize elements of C1 and C2 by dividing each element by filter size since there are 4 elements at
each filter (i.e. (2 × 2) = 4).
105
106
7 Convolutional Neural Networks
them by dividing each element by the number of elements used in the convolution filters. Since
there are four elements in each convolution filter, f (C1 ) and f (C2 ) elements are divided by 4, as
shown in Figure 7.12c. Since RELU only replaces negative values with zero, it does not change the
sizes of the matrices generated by the CL.
Pooling layer operations: As part of PL operations, a pooling size smaller than the RELU
matrices is selected, and pooling is conducted such that the maximum-valued RELU element
overlapping with the pooling size is recorded. Pooling continues in a sliding manner similar to
convolution until all RELU elements are covered. For this example, a pooling size of (2 × 2) is
selected, which is smaller than the RELU matrix size of (3 × 3). Stride value (i.e. the number
of cells to be used in sliding the pool) is selected as one. The pooling operation is illustrated in
Figure 7.13. After PL operations complete, C1 ′ and C2 ′ matrices are obtained from the RELU
matrices. Note that the matrices generated by the PL (i.e. C1 ′ and C2 ′ , each of size (2 × 2))
have smaller dimensions than the ones generated by the CL (i.e. C1 and C2 , each of size 3 × 3).
Reduction in dimensions depends on the pooling size and stride. Note that the convolution
numbers of the maximum elements are recorded to be used later when performing back
propagation.
Unrolling layer operations: The C1 and C2 matrices are unrolled into one-dimensional vectors,
as shown in Figures 7.14a and b. The size of the UL result is the total number of elements in the
C1 and C2 matrices. Afterwards, the vectors are concatenated into a single vector. For this example,
the C1 and C2 matrices have dimensions of (2 × 2) each. Therefore, the UL generates a vector with
eight elements, which will be used as the input layer of the fully-connected ANN, as shown in
Figure 7.14.
ANN operations: A fully-connected ANN will be receiving the one-dimensional vector
generated by the UL. For this example, there are eight elements in this vector, as shown in
Figure 7.13
PL operations, where a pooling size of (2 × 2) and a slide value of 1 are selected.
7.5 A Numerical Example of a CNN
Figure 7.14 Unrolling layer operations: (a) C1
matrix and (b) C2 matrix, and (c) combining both into
a single unrolled matrix.
(a)
(b)
(c)
Figure 7.15 Fully-connected ANN used in CNN for this example with (8 × 4) synapses between the inputs
and hidden layer (i.e. |W (1) | = 32) and (1 × 4) for bias (i.e. |B(1) | = 4), and (4 × 2) neurons between the
hidden layer and the output layer (i.e. |W (2) | = 8) and (1 × 2) for bias (i.e. |B(2) | = 2).
Figure 7.14c. Therefore there will be eight inputs in the ANN to be placed after the UL. For
simplicity, we select a fully-connected ANN with a four-neuron hidden layer (Figure 7.15). Since
there are two outputs for this example, one for shape O and the other for X, the output layer of the
ANN will have two neurons.
As shown in Figure 7.14c, the inputs and weights for the ANN layer are
[
]
X = 1.03 0.53 0.53 1.03 1.05 1.05 0.55 1.05 .
Since the example input is in the shape of an O (Figure 7.7), the outputs of the ANN will be Y1 = 1
and Y2 = 0. In matrix form, these outputs are
[ ]
Y= 1 0.
107
108
7 Convolutional Neural Networks
There are eight inputs generated by the UL to be fed into the ANN. Since there are four neurons
selected for the hidden layer, there are (8 × 4) synapses connecting the inputs to the hidden layer.
In addition, there will be a bias connected to each neuron of the hidden layer, represented as a
vector of (4 × 1) (see Chapter 2). Similarly, between the neurons of the hidden and output layers
there are (4 × 2) synapses and bias vector of (1 × 2). The back-propagation operation starting at
the ANN and flowing through the layers of the CNN is presented in Appendix C.
7.6 Computational Cost of CNN Training
Let us consider the computational cost of CNN training. Suppose the dimensions of a CNN input
are given as (di × hi × wi ) and the CL uses a total of Fcl filters, each with the size of (di × fcl × fcl ). As
presented in Figure 7.16, using a stride size of scl , after applying the convolution operation, the CL
will then generate Fcl matrices, each of size
(
) (
)
hi − fcl
wi − fcl
+1 ×
+1 .
scl
scl
Note that even though an input matrix may be di -dimensional, the CL always generates a 2-D
matrix. For typical applications, the number of convolutional filters Fcl in CL is selected between
32 and 512, the filter size fcl as an odd integer (so that the filter contains a central pixel) and the
stride size as scl = 1. Then, the size of each of Fcl output matrices becomes
(
) (
)
hi − fcl + 1 × wi − fcl + 1 .
(7.1)
Due to the nature of the convolution operation, a CL results in a matrix that is smaller in size
than its input. This is because the filter cannot cover input matrix elements near the borders,
resulting in information loss at the borders. A solution to remedy this shortcoming is zero
padding [77], where additional rows and columns of zeros are added to the input matrix. In
Figure 7.17a, convolution of an input matrix of size (4 × 4) using a (3 × 3) filter generates an output
matrix of (2 × 2). However, after adding zeros to surround the input matrix, an output matrix of
(3 × 3) is generated (Figure 7.17b). If necessary, with more padding, the output and input matrices
can be made the same size.
Figure 7.16
CNN convolutional layer – dimensions of input and output matrices.
7.6 Computational Cost of CNN Training
(a)
(b)
Figure 7.17 Zero padding prevents loss of information during convolution operation: (a) input image
generates a matrix with smaller dimensions, (b) by zero padding, information loss can be prevented (for no
information loss, input matrix must be padded to be size of (1 × 8 × 8) for this example).
After RELU, which does not change the dimensions of the matrices, PL with a pool size of p and
stride of spl generates Fcl matrices, each with the size of
⎞ ⎛ wi − fcl
⎞
⎛ hi − fcl
+1−p
+1−p
⎟ ⎜ s
⎟
⎜ s
cl
cl
⎜
+ 1⎟ × ⎜
+ 1⎟ .
⎟ ⎜
⎟
⎜
spl
spl
⎟ ⎜
⎟
⎜
⎠ ⎝
⎠
⎝
Pooling size p is typically selected such that wi and hi are divisible by p. Stride spl is set as spl = p
so that there is no overlap between different pooling regions. For scl = 1, as shown in Figure 7.18,
Figure 7.18
CNN pooling layer input and output matrix dimensions.
109
110
7 Convolutional Neural Networks
we have the dimensions of each Fcl matrix as
) (
)
(
wi − fcl + 1 − p
hi − fcl + 1 − p
+1 ×
+1 .
spl
spl
(7.2)
The size of the UL input is the total amount of data contained in all Fcl matrices generated by PL.
The output of the UL is also the number of inputs to the ANN:
⎛ hi − fcl
⎞ ⎛ wi − fcl + 1
⎞
+1−p
−p
⎜ s
⎟ ⎜
⎟
s
cl
cl
+ 1⎟ × ⎜
+ 1⎟ .
Iann = Fcl × ⎜
⎜
⎟ ⎜
⎟
spl
spl
⎜
⎟ ⎜
⎟
⎝
⎠ ⎝
⎠
As shown in Figures 7.19 and Figure 7.20, for scl = 1 and hi = wi , we have
(
)2
hi − fcl + 1 − p
+1 .
Iann = Fcl ×
spl
(7.3)
An additional function, called softmax, can be applied to the outputs of a CNN to convert
values of L to probabilities. The sum of all probabilities adds up to 1, where each output probability
Figure 7.19
CNN pooling layer output and unrolling layer input matrices.
Figure 7.20
CNN unrolling layer outputs as ANN inputs.
7.6 Computational Cost of CNN Training
represents how confident the CNN is that a particular output is the solution. The probability of an
output Yk is calculated as
P(Yk ) =
eYk
,
L
∑
Yi
i=1
where each probability is between 0 and 1 even for negative values of CNN outputs.
For a fully-connected ANN with a single hidden layer containing H neurons, the computational
cost of training will be (Iann × H) for one epoch, where  shows the order of magnitude of the
number of computational operations [35]. If there are multiple hidden layers, for each layer i with
Hi neurons, a computation load of (Hi−1 × Hi ) is added. For training a CNN for E epochs, the
computational cost is multiplied by E. In addition, if there are N input matrices used in a sample
set, the cost must also be multiplied by N.
In real-life applications, it is customary to employ several consecutive CL–PL pairs in a CNN to
improve its prediction ability and reduce the dimensions of the matrices before they are fed into the
ANN, where each additional CL–PL pair typically uses twice as many filters as the previous pair,
thereby reducing the matrix size by the order of (1∕spl ) while increasing the number of output
matrices by (fcl ). There are several mechanisms employed to reduce matrix sizes in neural networks, such as embedding layers (see Section 10.5 in Chapter 10). In Appendix C, we present
training steps for a CNN using a numerical example. We recommend the readers study the CNN
training process to appreciate its computational cost.
Let us now present a simple numerical example to examine the number of operations needed to
train a CNN. Suppose an image of (3 × 1000 × 1000) is used to train a CNN with 3 CL–PL pairs.
From Eq. (7.1), the first CL (let us call it CL-1), using F = 20 filters each of size fcl = 5, generates 20
output matrices, each of size (996 × 996). Based on Eq. (7.2), for the first PL (let us call it PL-1), using
a pool size of p = 3 and stride of s = 3, the output will be 20 matrices, each of size (332 × 332). At
CL-2, suppose there are F = 35 filters, each of size fcl = 5. Using the outputs of PL-1, CL-2 generates
35 output matrices, each of size (328 × 328). In turn, PL-2, with pool size of p = 3 and stride of
s = 3, generates 35 output matrices, each of size (109 × 109). Similarly, CL-3, using F = 50 filters,
each of size fcl = 5, generates 50 output matrices, each of size (105 × 105). PL-3, with pool size of
p = 3 and stride of s = 3, generates 50 output matrices, each of size (35 × 35). At this point the
output is fed into the UL, which will unroll the matrices and generate an input for the ANN of
size (61 250 × 1 × 1). Suppose the fully-connected ANN has three hidden layers, each with 1000
neurons. In this case, the number of operations for ANN training in forward propagation is
Tcnn = (61 250 × 1000) + (1000 × 1000) + (1000 × 1000) + (1000 × L) ≈ 63M,
where L is the number of CNN outputs (insignificant in the total, which is why it is not shown
in the estimate). With back-propagation, the number of operations are doubled to approximately
126M. For a typical ANN training with E = 1000 epochs, the total cost for one input image is 126B
operations. If there are 10 000 images in a moderate-sized sample set, the training cost adds up to
one quadrillion operations: Tcnn = (1015 ). Figure 7.21 illustrates this example.
The cost could be much higher if larger images, more CL–PL layer pairs and ANN hidden layers
with higher numbers of epochs are employed. Depending on the specific requirements of an
application, the data scientists and subject matter experts should find an optimal balance among
such parameters. Our goal by giving this numerical example here is to underline the fact that the
computational cost of training a CNN can easily be prohibitively high if special attention is not
paid to the characteristics of each CNN layer.
111
112
7 Convolutional Neural Networks
Figure 7.21 A numerical example for the computational cost of CNN training for an input image of
(3 × 1000 × 1000) with 3 CL-PL pairs and an ANN with 3 hidden layers, each with 1000 neurons; for 10 000
images and 1000 epochs, the expected computational cost is Tcnn = (1015 ) operations.
7.7 Concluding Remarks
In this chapter, we introduce the main concepts governing the operations and functionalities of
CNNs, which are ideal for the identification of multi-dimensional inputs such as images and
speech. A typical CNN consists of convolution, pooling and unrolling layers; rectilinear activation
units; and a fully-connected ANN. The main goal for engaging different types of layers is to
reduce the high dimensions of the inputs. Each layer reduces the input image size by the order
of the square of the filter sizes. Depending on the level of precision sought, multiple layers are
employed.
In general, training CNNs requires a high number of operations. One has to keep in mind their
effect on training cost when selecting filter sizes, the number of filters and CL–PL pairs, and ANN
architecture. It is highly recommended that techniques such as embedding are utilized to keep the
computational cost of training within feasible boundaries.
7.8 Exercises for Chapter 7
1) Discuss the impacts of increasing the number of CL–PL pairs on CNN training and performance.
2) Discuss the impacts of using large filter sizes on CNN training and performance.
3) Discuss the impacts of using small filter sizes on CNN training and performance.
4) Discuss the impacts of having an ANN with a large number of hidden layers on CNN training
and performance.
7.8 Exercises for Chapter 7
5) How will the changes suggested in Exercises 1 and 2 together impact CNN training and
performance? Can you support your answer by deriving matrix sizes as information is
processed through the CNN?
6) Repeat Exercise 5 for the changes suggested in Exercises 1 and 3.
7) Repeat Exercise 5 for the changes suggested in Exercises 1 and 4.
8) Repeat Exercise 5 for the changes suggested in Exercises 2 and 4.
113
115
8
CNNs for Optical Character Recognition
8.1 Introduction
Optical character recognition (OCR) is the process of digitizing and recognizing images of written,
typed or printed text [78]. A typical example of OCR usage is postal automation, where letters and
packages are classified based on their hand-written or typed addresses and labels. Rather than
manually reading and typing the address of each letter into a computer, OCR allows the computer
to recognize and process the addresses automatically. Other applications of OCR include capturing
licence plate numbers, conversion of physical documents to editable text documents and assistance
for the blind and visually impaired. A popular information source used for OCR is the MNIST
dataset [79]. It consists of 70 000 greyscale images of handwritten numerical digits ranging from
0 to 9. Each digit is stored as a (28x × 28) pixel image. The dataset is divided into 60 000 training
samples and 10 000 test samples. Figure 8.1a displays a few examples from the MNIST dataset [80].
The MNIST website contains the performance of various classification methods trained on the
dataset. It is reported on the MNIST website that CNNs currently outperform every other classifier
for character recognition applications. In Figure 8.1b, the lowest error rate corresponding to each
classifier is listed.
8.2 A Simple CNN for OCR
In this chapter, we introduce a CNN for implementing a simple OCR task, namely recognizing
several hand-written characters. The CNN in this example reads a hand-drawn greyscale image of
size (1 × 28 × 28) and identifies the input character as one of the reference images that it has stored.
Multiple consecutive CL–PL pairs are employed to increase the prediction capability of the CNN
(Figure 8.2). There are three convolutional layers (CLs), using Fcl = 8, 16 and 32 filters in three CLs,
each with a size of fcl = 3 × 3. There are three RELU entities and two pooling layers (PLs), each with
a pooling size of p = 2 using stride of spl = 2. The outputs of the ANN also constitute the output layer
of the CNN. The number of ANN outputs is the number of distinct characters that the CNN was
trained on. Based on the input image given to the CNN, only one of these CNN outputs will be set
to true (whichever output the CNN predicts as the correct match for the input image), while the
others will be set to false. Let us now examine the image dimensions as the input goes through
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
116
8 CNNs for Optical Character Recognition
(a)
(b)
Figure 8.1 Essential information for OCR applications in MNIST website: (a) sample images of digits and
(b) CNNs outperform all other OCR classifiers. Adapted from [77].
Figure 8.2
CNN example for the simple OCR application presented in this chapter.
the layers of the CNN for this example (Figure 8.3). Using Eqns. (7.1), (7.2) and (7.3) presented in
Chapter 7, we have:
Input: Image size is (1 × 28 × 28).
1st CL: Using Fcl = 8 filters, each with size of fcl = 3 × 3, dimensions are changed from (1 × 28 ×
28) to (8 × 28 × 28).
st PL: Using a pool size of p = 2 and stride s = 2, dimensions are reduced from (8 × 28 × 28)
● 1
pl
to (8 × 14 × 14).
nd CL: Using F = 16 filters (i.e. twice the previous number of filters), each with size f = 3 × 3,
● 2
cl
cl
dimensions are changed from (8 × 14 × 14) to (16 × 14 × 14).
nd PL: Using a pool size of p = 2 and stride s = 2, dimensions are reduced from (16 × 14 × 14)
● 2
pl
to (16 × 7 × 7).
nd CL: Using F = 32 filters (i.e. twice the previous number of filters), each with size of f =
● 3
cl
cl
3 × 3, dimensions are changed from (16 × 7 × 7) to (32 × 7 × 7).
● UL: Unrolling changes its input from (32 × 7 × 7) to (1568 × 1 × 1).
● ANN: This has one fully connected layer and output size of (C × 1 × 1), where C is the number
of unique characters to be identified by the CNN.
●
●
8.3 Organization of Training and Reference Files
Figure 8.3 Image dimensions as it is processed through CNN layers: An input image with size
(1 × 28 × 28) is fed to CNN; first CL changes size to (8 × 28 × 28), the first PL to (8 × 14 × 14), the second CL
to (16 × 14 × 14), the second PL to (16 × 7 × 7), the third CL to (32 × 7 × 7), and UL to (1568 × 1 × 1) before
feeding it to the ANN.
8.3 Organization of Training and Reference Files
Before introducing the scripts implementing a CNN, let us present a folder structure to store the
files used in training and prediction. Note that this organization is only meant to be illustrative
and may not be followed verbatim. Suppose we create two folders, named TrainingSet and
ReferenceSet, to hold training and reference images, respectively.
TrainingSet: This folder stores the image files used for training the CNN. There is one
sub-folder for each distinct character to be identified by the CNN. For an output layer with
dimensions of C × 1 × 1, there are C sub-folders below the TrainingSet folder, one for
each output character. In each sub-folder, there are image files to be used in training the CNN
for recognizing that particular image. An example for the TrainingSet folder is shown
in Figure 8.4, with five distinct images to be recognized by the CNN, named as eleven,
seven, six, umlaut, and w. For example, in sub-folder six, we see that there are five
different images to be used in training the CNN for this output, stored in files six_1.png to
six_5.png.
● ReferenceSet: Files under this folder contain the official (i.e. reference) images to be
displayed to a user after the CNN identifies an input image. It contains the same number
of sub-folders as TrainingSet, which is the number of outputs defined for the CNN. For
simplicity, we assumed that both folders have the same sub-folder names. When the CNN
identifies that an input image belongs to one of the images in sub-folder F in TrainingSet,
the corresponding image in sub-folder F under ReferenceSet will be displayed to the user.
●
117
118
8 CNNs for Optical Character Recognition
Figure 8.4 In this sample implementation, images used for training the CNN are stored in a folder called
TrainingSet, where there are as many folders as the number of characters to be recognized by the CNN,
and in each of the folders are the individual images associated with that character (in this example, a folder
called six contains all the images of the character 6 used in training, namely, the files called six_1.png
to six_5.png).
Figure 8.5 In this sample implementation, when an input image is identified, it is displayed using the
reference image stored in the corresponding sub-folder under ReferenceSet (in this example, once an
input is identified belonging to the images in sub-folder six in TrainingSet, the contents of
Ref_six.png in the sub-folder six but in ReferenceSet are displayed as the CNN output).
In Figure 8.5, the contents of sub-folder six in ReferenceSet are shown. If the CNN decides
that an input image looks similar to the images in sub-folder six under TrainingSet, then
the contents of Ref_six.png in sub-folder six under ReferenceSet are displayed to the
user as the output of the CNN.
8.4 Python Script for OCR Using a CNN
8.4 Python Script for OCR Using a CNN
A sample implementation of a CNN for OCR in Python is presented below. The user is given the
choice of either training a new CNN or using an already trained one in line 20. If the user enters
1 as the option, a new CNN is trained by calling the function CNN_Train (described below) and
the resulting object stored in model in line 26 as follows:
model = CNN_Train(train_dir)
where variable train_dir contains the address of the TrainingSet folder described in
Section 8.3, which will be used for training a new CNN. If the user enters 2 as the option,
a previously trained and saved CNN is loaded into the model object in line 26. In line 46, a
function named draw_tool is called, which returns the user-drawn image input to be located in
image_path. In line 47, a function named CNN_predict is called as follows:
CNN_predict(model, ref_dir, image_path)
where model contains the CNN (either newly trained or a previously saved one from above),
ref_dir is the location of the ReferenceSet folder (as explained in Section 8.3 above) and
image_path is the user-drawn input image.
In lines 55–105, a function called draw_tool is implemented. It presents the user with a canvas,
stores the drawn image into a file and returns a pointer to this file.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# The City College of New York, City University of New York
# Written by Olga Chsherbakova
# October, 2023
from CNN_Train import CNN_Train
from CNN_Predict import CNN_Predict
import os
from tensorflow.keras import models
import tkinter as tk
from PIL import Image, ImageDraw
import tempfile
# Optical character recognition using a CNN
def char_recognition_MAIN_noGUI():
# select training directory
train_dir = 'TrainingSet'
# select reference and output directories
ref_dir = 'ReferenceSet'
saved_cnns = 'Saved_CNNs'
print('WELCOME TO USING CNN. MAKE YOUR SELECTION:')
print('Enter 1 to train a new CNN')
print('Enter 2 to load an existing CNN')
user_entry = input('Enter your choice: ')
if user_entry == '1':
#train a new CNN:
print('Be patient until statistics are displayed...')
model = CNN_Train(train_dir)
input('CNN is trained. Press enter to use the CNN\n')
elif user_entry == '2':
# use an existing CNN:
cnn_file = input('Enter CNN name (include.h5 extension): ')
try:
# load the CNN and assign it to cnn_trained
model = models.load_model(os.path.join(saved_cnns,
cnn_file))
except:
print('Need a valid file name... Leaving... Bye')
return
else:
print('Need a valid choice... Leaving... Bye')
return
119
120
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
8 CNNs for Optical Character Recognition
# Now there is a trained CNN loaded (either a new one or
# an existing one). Use trained CNN to make predictions:
# start drawing a character
image_path = draw_tool()
CNN_Predict(model, ref_dir, image_path)
print('Have a predictably nice day!')
os.remove(image_path)
def draw_tool():
root = tk.Tk()
root.title("Draw a Character")
canvas_width = 280
canvas_height = 280
# Create a canvas for drawing
canvas = tk.Canvas(root, width=canvas_width,
height=canvas_height, bg="white")
canvas.pack(pady=20)
# Variables to store the last x and y positions
lastx, lasty = None, None
# Create an image to save
image = Image.new("RGB", (canvas_width, canvas_height),
"white")
draw = ImageDraw.Draw(image)
# Define filepath here
filepath = ""
def on_button_press(event):
nonlocal lastx, lasty
lastx, lasty = canvas.canvasx(event.x), canvas.canvasy(event.y)
def on_mouse_drag(event):
nonlocal lastx, lasty
x, y = canvas.canvasx(event.x), canvas.canvasy(event.y)
canvas.create_oval((lastx, lasty, x, y),
fill="black", width=10)
draw.line([lastx, lasty, x, y], fill="black", width=10)
lastx, lasty = x, y
def save_image():
nonlocal filepath
# Save to a temporary file
with tempfile.NamedTemporaryFile(delete=False,
suffix='.png') as tmp_file:
filepath = tmp_file.name
image.save(filepath)
root.destroy()
# Bind events to the canvas
canvas.bind("<Button-1>", on_button_press)
canvas.bind("<B1-Motion>", on_mouse_drag)
# Add a save button
save_button = tk.Button(root, text=
"Ckick here when finished", command=save_image,
bg="blue", fg="white")
save_button.pack(pady=20)
root.mainloop()
return filepath
char_recognition_MAIN_noGUI()
Below we present a function named CNN_Train. In lines 26–35, training images are loaded
into an object named train_generator. Between lines 38 and 81, the topology for the CNN is
defined. In line 41, an input layer of size (28 × 28 × 1) is defined:
layers.InputLayer(input_shape = (28,28,1))
8.4 Python Script for OCR Using a CNN
In line 46, the first CL is created with 8 filters, each of which with the size of (3 × 3), using padding
so that the input and output images will have the same dimensions (i.e. without losing any of data
points due to the convolution operation):
layers.Conv2D(8,(3,3), padding = 'same')
In line 50, the CL output is normalized to have values between −1 and 1:
layers.BatchNormalization()
In line 53, a RELU is defined as follows:
layers.Activation('relu';)
In line 56, a PL with a pooling size of (2 × 2) and a stride of 2 is defined by
layers.MaxPooling2D((2,2), strides=2)
In lines 61–63, a second CL is defined with 16 filters, each with the size of (3 × 3), a normalization
layer and a RELU. A second PL with pooling size of (2 × 2) and stride of 2 is defined in line 66.
Similarly, in lines 71 to 74, the third CL with 32 filters, a normalization layer and a RELU are
defined. Line 75 shows the unrolling layer as follows:
layers.Flatten()
In lines 76 to 80, a fully-connected ANN using softmax as its activation function is defined as
follows:
layers.Dense(train_generator.num_classes, activation = 'softmax')
After defining training options in lines 89–91 as using gradient descent for the optimization
method (i.e. option Adam) and selecting a cross-entropy loss function to compute losses at each
epoch, the CNN is trained for 700 epochs in line 93:
model.fit(train_generator, epochs=700)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import tensorflow as tf
from tensorflow.keras import layers, models, optimizers
import os
def count_files_in_subdirectories(root_dir="TrainingSet"):
sum_files = 0
# Get the list of the all files in the folders
subdirs = [d for d in os.listdir(root_dir) \
if os.path.isdir(os.path.join(root_dir, d))]
# Itterate in each of the folder
for subdir in subdirs:
subdir_path = os.path.join(root_dir, subdir)
files_count = len([f for f in os.listdir(subdir_path) \
if os.path.isfile(os.path.join(subdir_path, f))])
sum_files += files_count
return sum_files
# Train a new CNN using information from TrainingSet directory
# Then return trained CNN
# TRAIN THE CNN
def CNN_Train(train_dir):
# load the training image dataset into an ImageDatasGenerator
# train_path contains the path to the dataset
train_datagen = \
tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(28, 28),
# get the number of unique training labels in the training data
121
122
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
8 CNNs for Optical Character Recognition
batch_size=count_files_in_subdirectories(),
class_mode='categorical',
color_mode='grayscale'
)
# Define the CNN structure
model = models.Sequential([
# set the size of the input image as 28x28 pixels with 1 color
# channel (grayscale)
layers.InputLayer(input_shape=(28, 28, 1)),
# create the first convolution layer, it has 8 convolutional
# filters, each with a height and width of 3 and input is 0 padded.
# the input is padded with the appropriate number of zeros so that
# the size of the output is the same size as the input.
layers.Conv2D(8, (3, 3), padding='same'),
# normalize the output of the convolution between the range -1
# and 1. this prevents the updated weights from increasing
# explosively during training.
layers.BatchNormalization(),
# apply rectilinear unit activation function that sets all negative
# values to zero
layers.Activation('relu'),
# apply max pooling using a pooling size of (2x2), and using a
# stride of 2
layers.MaxPooling2D((2, 2), strides=2),
# create a second convolution layer, it has 16 convolutional
# filters, each filter has a height and width of 3 and the input is
# zero padded.
layers.Conv2D(16, (3, 3), padding='same'),
layers.BatchNormalization(),
layers.Activation('relu'),
# apply max pooling using a pooling size of (2x2), and using a
# stride of 2
layers.MaxPooling2D((2, 2), strides=2),
# create a third convolution layer, it has 32 convolutional
# filters, each filter has a height and width of 3 and the input is
# zero padded.
layers.Conv2D(32, (3, 3), padding='same'),
layers.BatchNormalization(),
layers.Activation('relu'),
layers.Flatten(),
layers.Dense(train_generator.num_classes,
# use a softmax layer to convert the outputs to a set of
# probalities where each output represents the
# probability that the image is a corresponding label.
activation='softmax')
])
# END OF layers
# optimizer to be used during training. In this case,
# the Adam optimizer is used. The optimizer is responsible for updating
# the model's weights during training based on the computed gradients.
# Adam is a popular optimization algorithm that adapts the learning
# rate during training.
model.compile(optimizer=optimizers.Adam(),
loss='categorical_crossentropy',
metrics=['accuracy'])
# specify CNN training options
model.fit(train_generator, epochs=700)
return model
A sample implementation of the CNN_Predict function is presented below. In line 61, the
trained CNN is run with the user-drawn image to generate an output:
predictions = model.predict(char_img_array)
which is then displayed in lines 62 to 70.
8.4 Python Script for OCR Using a CNN
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
import numpy as np
import os
from PIL import Image
from tensorflow.keras.preprocessing import image as keras_image
from PIL import Image, ImageTk
import tkinter as tk
from tkinter import filedialog
# this GUI shows prediction for the character drawn by the user
def display_images_tkinter(model,char_img_path, ref_image_path):
root = tk.Tk()
root.title("Character Recognition Using CNN")
# Load the images
char_img = Image.open(char_img_path).convert('L')
ref_img = Image.open(ref_image_path).convert('L')
# Resize the prediction image
ref_img = ref_img.resize((250,250))
char_img_tk = ImageTk.PhotoImage(char_img)
ref_img_tk = ImageTk.PhotoImage(ref_img)
# Create labels for the images
char_label = tk.Label(root, image=char_img_tk)
char_label.pack(side=tk.LEFT, padx=10)
ref_label = tk.Label(root, image=ref_img_tk)
ref_label.pack(side=tk.LEFT, padx=10)
def save_and_quit():
file_name = filedialog.asksaveasfilename(defaultextension=".h5",\
filetypes=[("H5 files", "*.h5")])
if file_name:
model.save(file_name)
# root.mainloop()
root.destroy()
def quit_root():
root.destroy()
# Button to save the model and quit
save_quit_button = tk.Button(root, text="Save Model and Exit", \
command=save_and_quit)
save_quit_button.pack(pady=20)
save_quit_button = tk.Button(root, text="Exit", command=quit_root)
save_quit_button.pack(pady=20)
root.mainloop()
def CNN_Predict(model, ref_path, char_img_path):
# Load and preprocess the image
char_img = keras_image.load_img(char_img_path, target_size=(28, 28),\
color_mode='grayscale')
char_img_array = keras_image.img_to_array(char_img)
char_img_array = np.expand_dims(char_img_array, axis=0)
char_img_array /= 255.0
# Make a prediction
predictions = model.predict(char_img_array)
predicted_class = np.argmax(predictions[0])
class_names = sorted(os.listdir(ref_path))
predicted_class_name = class_names[predicted_class]
# Retrieve the appropriate reference image for prediction
ref_image_path = os.path.join(ref_path, predicted_class_name, \
os.listdir(os.path.join(ref_path, predicted_class_name))[0])
display_images_tkinter(model,char_img_path, ref_image_path)
123
124
8 CNNs for Optical Character Recognition
(a)
(b)
Figure 8.6 Sample run of the Python script given in Section 8.4, where a trained CNN predicts and
displays a user drawn character: (a) user input, and (b) CNN prediction using the corresponding stored
reference image.
A sample run of the above Python script is shown in Figure 8.6. For a user-drawn character shown
in Figure 8.6a, the prediction generated by the Python script is displayed in Figure 8.6b using the
stored image in the ReferenceSet folder as explained above in Section 8.3.
8.5 Matlab Script for OCR Using a CNN
Before running the Matlab script, TrainingSet and ReferenceSet folders and their
sub-folders must be created as described above in Section 8.3. The sub-folders may be populated
by running the create _new_ character.m function shown below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
% create_new_character function used for training and reference folder images:
clc;
clear;
if exist('create_character_data_GUI','dir') == 7
% get access to all files in the 'create_character_data_GUI' directory
addpath('create_character_data_GUI');
else
error("Error: Unable to locate 'create_character_data_GUI' directory")
end
if exist('functions','dir') == 7
% get access to all files in the 'functions' directory
addpath('functions');
else
error("Error: Unable to locate 'functions' directory")
end
% open the welcome GUI
GUI_1_WELCOME()
The images may be used either as references or for training the CNN. The user is asked to draw
a character with the help of a mouse or writing pad (multiple strokes for drawing an image are
allowed). If the drawn image will be used as a reference image, the user should save it in ReferenceSet under the appropriate sub-folder. Similarly, if it will be used as a training sample image, it
should be saved in TrainingSet under the corresponding sub-folder. Figure 8.7 shows a sample
for creation of reference and sample training images using create_new_character.m, where
the user clicks on Draw a Character, draws a character and decides to save it in a sub-folder
either under TrainingSet or ReferenceSet.
8.5 Matlab Script for OCR Using a CNN
(a)
(b)
(c)
(d)
Figure 8.7 Creation of reference and sample training images using create_new_character.m:
(a) after user clicks on Draw a Character, (b) user draws a character, (c) user decides to save it, and
(d) user saves the image in a sub-folder either under TrainingSet or ReferenceSet (for both cases,
sub-folder names should be the same, for example umlaut).
A Matlab script implementing a CNN for OCR is presented below. Its structure is similar to the
Python script in Section 8.4 for ease of comparison. In the main program, the user is prompted to
create a new CNN or load an existing one. If the user chooses to train a new CNN by entering 1 in
line 23, this invokes CNN_Train in line 26:
cnn_trained = CNN_Train(train_dir);
where variable train_dir contains the address of TrainingSet folder, where the images to
train the CNN resides. If the user chooses to use a previously trained CNN by entering 2 as the
option, it loads an existing CNN from SavedCNNs directory (saved previously as a .mat file) in
line 36. In line 48, a function named draw_tool is called, which asks the user to create an input
image and saves it in an object called image. In line 51, CNN_Predict is called with the trained
CNN and the user input image:
cnn_prediction = CNN_Prediction(cnn_trained,image);
Between lines 50 and 60, the user is then asked whether new predictions using the same CNN
will be needed; if so, the above steps are repeated. The user is finally asked if the trained CNN is to
be saved for future use.
125
126
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
8 CNNs for Optical Character Recognition
% The City College of New York, City University of New York
% Written by Ricardo Valdez
% August, 2020
% Optical character recognition using CNNs
clc;
clear;
% select training directory
train_dir = 'TrainingSet';
% select reference directory
ref_dir = 'ReferenceSet';
%output directory
saved_cnns = 'Saved_CNNs';
if exist('functions','dir') == 7
% get access to all files in the 'functions' directory
addpath('functions');
else
error("Error: Unable to locate 'functions' directory")
end
message = sprintf('WELCOME TO USING CNN. MAKE YOUR SELECTION:');
disp(message);
disp('Enter 1 to train a new CNN 2 to load an existing CNN');
user_entry = input('Enter your choice: ');
if (user_entry == 1)
% train a new CNN:
disp('Be patient until statistics are displayed...');
cnn_trained = CNN_Train(train_dir);
input('CNN training is complete. Press enter to start using CNN.\n');
elseif (user_entry == 2)
% use an existing CNN:
cnn_file = input('Enter CNN name (dont forget .mat extension): ', 's');
cnn_path = pwd;
disp(cnn_file);
try
% load the CNN and assign it to cnn_trained
load_data = load(fullfile(cnn_path, saved_cnns, cnn_file));
cnn_trained = load_data.cnn_trained;
catch
disp('Enter a valid file name next time... Leaving... Bye');
return;
end
else
disp('Enter a valid choice next time... Leaving... Bye');
return;
end
% Use trained CNN to make predictions:
% start drawing a character
image = draw_tool();
% CNN predicts the character
cnn_prediction = CNN_Predict(cnn_trained, ref_dir, image);
user_entry = input('Do you want to save CNN? (y/n) ','s');
if(user_entry == 'y')
file_name = input('Enter file name (do not forget.mat extension): ',...
's');
save(fullfile(saved_cnns,file_name), 'cnn_trained');
else
end
disp('Have a predictably nice day!');
Below we present the function named CNN_Train, which defines the architecture of the CNN
in lines 16–66 as follows. In line 25, the first CL with its filter size of fcl = 3 and number of filters
of Fcl = 8 is defined. As discussed in Chapter 7, padding is set so that zeros are added to the input
image to keep it the same size after convolution, to avoid losing information due to the smaller size
of the CL filter fcl . After batch normalization in line 29, RELU is defined in line 32 and the first
PL is set with its size of p = 2 and stride of spl = 2 in line 36. The second CL–PL pair is defined in
lines 41 and 47. Then the third CL with its size, number of filters, RELU and normalization are
8.5 Matlab Script for OCR Using a CNN
in lines 51 to 53. Finally the fully-connected ANN is defined in line 58. Note that in Matlab, the
UL is implemented automatically. In lines 71–73, the options for training are defined as sgdm (i.e.
stochastic gradient descent), a learning rate of 0.01 and maximum number of epochs of 50. The
trained CNN is saved in an object named net in line 82.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
function net = CNN_Train(train_dir)
%
% Train a new CNN using information from TrainingSet directory
% Then return trained CNN
%%%%%%%%%%% TRAIN THE CNN %%%%%%%%%%%
% load the training image dataset into an imageDatastore
% train_path contains the path to the dataset
imds_train = imageDatastore(train_dir, ...
'IncludeSubfolders',true,...
'LabelSource','foldernames');
% get the number of unique training labels in the training data
noof_labels = height(countEachLabel(imds_train));
layers = [
% set the size of the input image as 28x28 pixels with 1 color
% channel (grayscale)
imageInputLayer([28 28 1])
% create the first convolution layer, it has 8 convolutional
% filters, each with a height and width of 3 and input is 0 padded.
% the input is padded with the appropriate number of zeros so that
% the size of the output is the same size as the input.
convolution2dLayer(3,8,'Padding','same')
% normalize the output of the convolution between the range -1
% and 1. this prevents the updated weights from increasing
% explosively during training.
batchNormalizationLayer
% apply rectilinear unit activation function that sets all negative
% values to zero
reluLayer
% apply max pooling using a pooling size of (2x2), and using a
% stride of 2
maxPooling2dLayer(2,'Stride',2)
% create a second convolution layer, it has 16 convolutional
% filters, each filter has a height and width of 3 and the input is
% zero padded.
convolution2dLayer(3,16,'Padding','same')
batchNormalizationLayer
reluLayer
% apply max pooling using a pooling size of (2x2), and using a
% stride of
maxPooling2dLayer(2,'Stride',2)
% create a third convolution layer, it has 32 convolutional
% filters, each filter has a height and width of 3 and the input is
% zero padded.
convolution2dLayer(3,32,'Padding','same')
batchNormalizationLayer
reluLayer
% create the ouput layer
% the number of neurons in the output layer is equal to the number
% of image labels
fullyConnectedLayer(noof_labels)
% use a softmax layer to convert the outputs to a set of
% probalities where each output represents the probability that the
% image is a corresponding label.
softmaxLayer
% add a classification layer to make each output mutually
% exclusive, in other words there's only a single correct label for
127
128
8 CNNs for Optical Character Recognition
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
% each image
classificationLayer];
%END OF layers
% specify CNN training options
options = trainingOptions('sgdm', ...
'InitialLearnRate',0.01, ...
'MaxEpochs',500);
% sleep for a second to display the GUI
pause(1);
% start training the CNN
% imds_train is imageDatastore that holds all training images and labels
% layers is the architecture of the CNN
% options contains the training parameters
net = trainNetwork(imds_train,layers,options);
end
Now we present the function named CNN_Predict with three input parameters, named net,
ref_path and char_img, which are pointers to the trained CNN, the address of the ReferenceSet folder and the user drawn input image, respectively. The trained CNN generates an
output in line 19 as follows:
[Prediction, ] = classify(net, char_img);
where Prediction contains an index to one of the sub-folders of the ReferenceSet folder. In
lines 22–33, the reference output image, which is pointed to by the top element of Prediction,
is displayed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
function Prediction = CNN_Predict(net,ref_path,char_img)
% this GUI shows prediction for the character drawn by the user
S.net = net;
S.ref = ref_path;
% open a window(figure) for the GUI
S.fh = figure('units','pixels',...
'position',[500 300 600 400],...
'menubar','none',...
'name','Character Recognition Using CNN',...
'numbertitle','off',...
'resize','off',...
'Color',[0.8588 0.9412 0.8275]);
% create an axes to hold the images
S.ax = axes('units','pix',...
'position',[100 250 200 90]);
% pass the drawing to the CNN and get a prediction
[Prediction,∼] = classify(net, char_img);
% retreive the appropriate reference image for prediction
full_ref_path = fullfile(ref_path,char(Prediction));
ref_imd = imageDatastore(full_ref_path);
% display user input image
subplot(1,2,1);
imshow(char_img);
title('YOUR CHARACTER','FontSize',15);
% display prediction image
subplot(1,2,2);
imshow(ref_imd.Files{1});
title('CNN PREDICTION','FontSize',15);
end
8.5 Matlab Script for OCR Using a CNN
The function named draw_tool is given below; this is used to display a simplistic graphical
interface for the user to draw a character.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
function image = draw_tool()
% draw_tool() displays a GUI that allows the user to draw a character.
% open a window(figure) for the draw tool
S.fh = figure('units','pixels', 'position',[500 300 600 400],...
'menubar','none', 'name','Character Recognition Using CNN',...
'numbertitle','off', 'resize','off','Color',[0.8588 0.9412 0.8275]);
% create a textbox
S.tb = uicontrol('style','text',...
'unit','pix', 'position',[15 360 570 30],...
'min',0,'max',2, 'fontsize',18,...
'string','DRAW A CHARACTER BELOW USING A MOUSE',...
'BackgroundColor',[1.0000 0.9490 0.7412],...
'fontweight', 'bold');
% create a blank axes
% this is where the user draw their character
S.ax = axes('units','pix',...
'position',[50 60 500 290],...
'XColor','none',...
'YColor','none');
% create a button to stop drawing
S.pb = uicontrol('style','push',...
'units','pix',...
'position',[100 10 400 40],...
'fontsize',14,...
'string','Click here when finished',...
'backgroundColor',[0.3020 0.4471 0.7412],...
'foregroundcolor','w',...
'callback',{@pb_call,S});
% set color for the drawing to blue
draw_color = [0 0.447 0.741];
colororder(draw_color);
% done is a boolean that indicates whether or not
% the user is done drawing their character
done = false;
first = true;
% draw a character
while(∼done)
% check if the GUI window is still open
if ishandle(S.fh)
% convert the drawing into a 28x28 image
if first == false
image = process_image(S.ax);
end
% draw a curve
get_curve(S.fh);
first = false;
else
done = true;
end
end
end
function [] = pb_call(varargin)
% button callback function
S = varargin{3};
% close the draw tool GUI
close(S.fh);
end
A sample run of the CNN implemented in Matlab above is shown in Figures 8.8a and b,
where a CNN is trained and the user is asked to draw a character. The input is then predicted
by CNN_Predict, and both the user input and the corresponding reference image from the
ReferenceSet folder are displayed.
129
130
8 CNNs for Optical Character Recognition
(a)
(b)
Figure 8.8 A sample run of the Matlab scripts given in Section 8.5: (a) a CNN is trained and the user is
asked to draw a character, and (b) the user input is fed into the trained CNN on the left, which first identifies
the image and displays the corresponding reference image from the ReferenceSet folder on the right.
8.6 Concluding Remarks
In this chapter, we introduce a CNN for the OCR class of problems, where a set of input image
samples are used to train a CNN. OCR is widely used in applications such as postal automation,
capturing licence plate numbers, conversion of physical documents to editable text files and assistance for the blind and visually impaired. We cannot employ ANNs for OCR applications since they
would be overwhelmed if multi-dimensional images with high pixel counts were directly fed into
them, which makes CNNs the most suitable AI tools for OCR applications. A CNN architecture
designed for OCR contains multiple CL–PL pairs to improve prediction capabilities. The designers have to be mindful of the computational cost of training when constructing a CNN topology
since, especially with a high number of samples, it could easily make a realistic project infeasibly
expensive.
8.7 Exercises for Chapter 8
1) Implement a CNN in Python similar to the one given in Section 8.4 to recognize the following
words: yes, no, up, down. User input will be written using Japanese characters. Display
the result in a user-selected language: (1) Japanese, (2) Polish, (3) Spanish, (4) Filipino, (5)
Swedish, and (6) English.
2) For the example in Exercise 1, if you increase the number of CL–PL pairs in the CNN, how do
you think this change will impact the CNN prediction performance? For each additional CL–PL
pair, can you detect a difference in time for training on your own processor?
8.7 Exercises for Chapter 8
3) For the example in Exercise 1, if you keep the CL–PL pairs the same but increase the pool and
filter sizes, how do you think this change will impact the CNN prediction performance? How
much do you have to increase their sizes until you can detect a difference in training time on
your own processor?
4) Repeat to Exercise 1 for the input language of Polish.
5) Repeat to Exercise 1 for the input language of Spanish.
6) Repeat to Exercise 1 for the input language of Filipino.
7) Repeat to Exercise 1 for the input language of Swedish.
8) Repeat Exercise 1 for a CNN implemented in Matlab similar to the one presented in Section 8.5.
131
133
9
CNNs for Speech Recognition
9.1 Introduction
In this chapter, we introduce a simple CNN implementation for speech recognition, which is the
ability for a computer program to recognize speech or recorded audio in any format. In typical
applications, recognized and identified speech will most likely be converted to another format
such as text or a digital record, thereby significantly increasing the influence of speech recognition
in the daily usage of various intelligent systems. Many modern devices have built-in speech
recognition capabilities through virtual assistant applications such as Alexa from Amazon and
Siri from Apple [81]. Other popular speech recognition applications include customer service
systems used by many commercial entities ranging from car rental companies to credit card
services, automatic generation of subtitles, machine translation tasks, assistance for the disabled
and background noise elimination in loud environments (e.g. noise cancellation devices used in
helicopters or submarine vessels).
A spectrogram [82] is a diagram that plots the spectrum of frequencies in an audio sample over
time. The spectrogram is calculated by using the Fourier transform to identify the frequencies
present in an audio clip in a given time window [83]. The pixel width of a spectrogram represents the number of time divisions in an audio file. For each time division, the Fourier transform is
calculated, producing a column of pixels that show the spectrum of frequencies during that sliver
of time. Pixels with high values signify that a certain frequency is active in the audio clip during that moment in time. An example spectrogram is shown in Figure 9.1 for an audio sample,
where each black pixel represents the frequencies present in the audio at certain times [84]. In a
spectrum, the y axis represents different frequency values, whereas the x axis shows the elapsed
time. In Figure 9.1, the frequency values are between 0 and 104 Hz, while the time extends from
0 to 0.836 seconds.
In this chapter, we introduce Python and Matlab scripts to implement CNN architectures to
recognize a specific a set of spoken words and translate them into different languages. Although
it is rather simplistic, the example CNN presented in this chapter illustrates the basic principles of employing CNNs for the speech recognition class of problems. Readers are encouraged
to experiment with CNN parameters such as the number and types of layers, filter quantities and sizes, and the number of epochs for training and observe their impact on prediction
accuracy.
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
134
9 CNNs for Speech Recognition
Figure 9.1 A speech spectrogram; each black pixel represents the frequencies present in the audio at
certain times.
9.2 A Simple CNN for Speech Recognition
Since audio signals can be represented as digital information in a spectrogram, they can be used
as inputs to a CNN. Similar methods to those used for character recognition by CNNs introduced
in Chapter 8 can now be employed for speech recognition. The CNN in this example reads a
spectrogram image of a spoken word with size (1 × 40 × 198) and identifies the spoken word input
as one of the stored reference audio files. Similar to the example CNN presented in Chapter 8 for
OCR applications, multiple CL–PL pairs are employed for speech recognition to reduce the size of
the spectrogram input (i.e. to down-sample the input) and to improve the CNN’s predictive power.
In this way, the CNN will work with smaller sized inputs, which in turn will reduce the time and
Figure 9.2 A typical CNN architecture example for speech recognition applications (multiple layers are
used in the CNN to reduce the size of the spectrogram input so that training is faster and over-fitting is
reduced).
9.2 A Simple CNN for Speech Recognition
computation cost required for training the CNN and enable inclusion of a wide variety of inputs
into its training set.
As shown in Figure 9.2, our example CNN for speech recognition consists of four CL–PL pairs,
one single CL and five ReLU entities. The fourth and final PL feeds into a fully-connected ANN
through a UL. There are L outputs for the ANN, which correspond to the number of unique words
that the CNN was trained on. Only one of those ANN L outputs will be selected as true for a
given input spectrogram, while the remaining outputs will be false. The ANN outputs are also
the outputs of the CNN. If a softmax layer is adopted [85], the CNN outputs will be converted to
probabilities rather than selecting one out of the L outputs. The dimensions of an input spectrogram
change as it is processed through the various layers of a CNN. Using Eqs. (7.1) – (7.3) presented in
Chapter 7, and employing zero padding in the CLs, the dimensions change at each CNN layer as
follows (Figure 9.3):
Figure 9.3 Spectrogram dimensions as it is processed through CNN layers: The input image size
of (1 × 40 × 198) is fed into CNN; first CL changes the image size to (12 × 40 × 198), the first PL
to (12 × 20 × 99), the second CL to (24 × 20 × 99), the second PL to (24 × 10 × 50), the third CL to
(48 × 10 × 50), the third PL to (48 × 5 × 25), the fourth and fifth CLs to (48 × 5 × 25), the fourth PL to
(48 × 5 × 13) and the UL to (3, 120 × 1 × 1) before feeding to the ANN.
135
136
9 CNNs for Speech Recognition
Input: Spectrogram of user audio input with size (1 × 40 × 198).
1st CL: With Fcl = 12 filters (typical for audio applications in Python and Matlab), each of size
fcl = 3 × 3, input spectrogram dimensions change from (1 × 40 × 198) to (12 × 40 × 198) (using
zero padding).
st PL: Using a pool size of p = (3 × 3) and stride of s = 2, spectrogram dimensions are further
● 1
pl
reduced from (12 × 40 × 198) to (12 × 20 × 99).
nd CL: There are F = 24 filters (twice the number of filters in the previous layer), each of size
● 2
cl
fcl = 3 × 3; spectrogram dimensions are changed from (12 × 20 × 99) to (24 × 20 × 99).
nd PL: Using a pool size of p = 3 × 3 and stride s = 2, spectrogram dimensions are reduced
● 2
pl
from (24 × 20 × 99) to (24 × 10 × 50).
rd CL: There are F = 48 filters (twice the number of filters in the previous layer), each of size
● 3
cl
fcl = 3 × 3; spectrogram dimensions are changed from (24 × 10 × 50) to (48 × 10 × 50).
rd PL: Using a pool size of p = 3 × 3 and stride of s = 2, spectrogram dimensions are reduced
● 3
pl
from (48 × 10 × 50) to (48 × 5 × 25).
th CL: There are F = 48 filters (same as previous CL), each of size f = 3 × 3; spectrogram
● 4
cl
cl
dimensions remain at (48 × 5 × 25).
th CL: There are F = 48 filters (same as previous CL), each of size f = 3 × 3; spectrogram
● 5
cl
cl
dimensions remain at (48 × 5 × 25).
th PL: Using a pool size of p = 1 × 13 and stride of s = 1, spectrogram dimensions are reduced
● 4
pl
from (48 × 5 × 25) to (48 × 5 × 13).
● UL: After unrolling, spectrogram dimensions are changed from (48 × 5 × 13) to (3, 127 × 1 × 1)
● ANN: Using a fully-connected layer and no hidden layers, input and output sizes are 3127 and
(L × 1 × 1), respectively, where L is the number of unique words to predict.
●
●
9.3 Organization of Training and Reference Files
Similar to the OCR implementation presented in Chapter 8, a directory structure is assumed to
be present in the host processor running the scripts. Two folders, named TrainingSet and
ReferenceSet, are to be created to store the spectrogram files used for training and reference
purposes, respectively. Figures 9.4 and 9.5 show a sample organization for the training and
reference audio files, which is only meant to be an illustrative design and may not to be followed
verbatim. Contents of the TrainingSet and ReferenceSet folders are summarized as follows:
TrainingSet folder: The audio files to be used for training the CNN are stored in this folder.
For each distinct word, there is one sub-folder containing the training files for that word. For
a CNN with L outputs, there are L sub-folders under the TrainingSet folder, one for each
output word. For example, in the TrainingSet folder shown in Figure 9.4, there are six different sub-folders, namely Goodbye, Hello, Nazdarovya, Sagol, SunnySideUp and
Welcome, which are the names of the words to be recognized by this example CNN. In each
sub-folder, there are audio files to be used in training for this word. Note that if multiple language inputs are to be accepted by this CNN, then all samples corresponding to the same word
will be placed into the same sub-folder. For example, in sub-folder SunnySideUp, there are
audio files for English, Korean and Polish pronunciations of the word(s) “sunny side up”
(there are several files for each language).
● ReferenceSet folder: In this folder, there are the same number of sub-folders as in TrainingSet (for simplicity, with the same names). When the trained CNN identifies a spoken input
●
9.3 Organization of Training and Reference Files
Figure 9.4 Audio files stored in the TrainingSet folder are used to train the CNN, where there are as
many sub-folders as the number of distinct words to be recognized by this CNN; within each sub-folder are
placed individual audio files belonging to that word (in this example, under the SunnySideUp sub-folder
there are audio files in English, Korean and Polish for the word(s) “sunny side up”.
word contained in a sub-folder in TrainingSet, the audio file with the reference pronunciation
is replayed, which is located in ReferenceSet, in a sub-folder with the same name. When a
different language pronunciation is desired, appropriate sub-folders for each language are placed
under ReferenceSet. For example, in Figure 9.5, a spoken word can be translated into four
different languages, namely Arabic, Korean, Polish and Spanish, each of which has
Figure 9.5 The ReferenceSet folder has sub-folders corresponding to the different languages to be
translated; within each sub-folder there are sub-folders as in TrainingSet (see Figure 9.4). When the
user pronounces “goodbye” and wishes it to be translated into Korean, the CNN first identifies goodbye as
it belongs to the first sub-folder in TrainingSet, then identifies the sub-folder corresponding to Korean
in ReferenceSet and replays the reference audio file called Ref_GoodBye_in_Korean.wav in the
first sub-folder under Korean.
137
138
9 CNNs for Speech Recognition
a dedicated sub-folder under ReferenceSet. The reference audio files for each distinct output are placed in the same order as in TrainingSet within each language sub-folder. Based
on user selection of a target language, the CNN will identify the spoken word and replay the
translation (i.e. the audio file) in the desired language placed in the corresponding sub-folder.
Suppose user input of “goodbye” is to be translated into Korean. First a trained CNN identifies the spoken word as it belongs to the first sub-folder in TrainingSet, then identifies
the sub-folder corresponding to Korean in ReferenceSet and replays the audio file named
Ref_GoodBye_in_Korean.wav.
9.4 Python Script for Speech Recognition Using a CNN
In this section, we present a sample Python script implementation of a CNN designed for speech
recognition in Figure 9.2. Before the Python script presented here is run, audio files for training
and reference pronunciations are assumed to have been created and stored in TrainingSet and
ReferenceSet folders and their sub-folders as defined in Section 9.3.
A sample Python script for speech recognition is presented below. Between lines 18 and 33 is
a utility function named rename_wav_file which renames a given file. Between lines 35 and
120, the main portion of the Python script is presented. The user is given an option to either train
a new CNN or use a previously trained one in line 44. If the user enters 1 as a choice, a new CNN
is trained in line 49:
cnn_trained, audio_files = CNN_Train(train_dir)
where the trained CNN is saved in cnn_trained object, and train_dir points to TrainingSet folder containing the audio files used for training the CNN. If the user chooses to use
a saved CNN by entering 2, the CNN and training files are set in lines 56 and 57:
cnn_trained = models.load_model(cnn_file)
audio_files = get_audio_files(train_dir)
where cnn_trained has the CNN object and audio_files is a pointer to the TrainingSet
folder. In lines 75–77, the user speaks a word and it is recorded in an object named rec_data.
In lines 85–97, the target language for CNN output is determined based on user input. The CNN
generates a prediction in line 100:
cnn_prediction = CNN_Predict(rec_data,…, cnn_trained, audio_files)
where cnn_prediction contains the output generated by the CNN, which is played back to the
user in line 110:
Playback_translation = CNN_Predict(rec_data,…, full_ref_path)
where full_ref_path contains the pointer to the reference audio files in the ReferenceSet
folder set up in lines 103–106. Finally, the user is asked to save the trained CNN in lines 115 and 117.
1
2
3
4
5
6
7
# The City College of New York, City University of New York
# Written by Olga Chsherbakova
# October, 2023
#
# Speech Recognition and Translator System using CNNs
import os
9.4 Python Script for Speech Recognition Using a CNN
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
import sounddevice as sd
from CNN_Predict import CNN_Predict
from PLAYBack_Translation import PlayBack_Translation
from CNN_Train import CNN_Train,get_audio_files
from keras import models
# This function is designed to rename a specific type of WAV file, assuming a
# particular naming convention where "ref.wav" is removed from the original
# filename. It is useful when you have multiple WAV files following this
# naming pattern and want to create new filenames with "ref.wav" removed.
def rename_wav_file(input_path):
try:
path, filename = os.path.split(input_path)
_, file_extension = os.path.splitext(filename)
# Create a new file name ("you_ref.wav")
new_filename = filename.split("_")
new_filename = "_".join(new_filename[:-1]) + "_ref.wav"
# Create new path
output_path = os.path.join(path, new_filename)
return "\\".join(output_path.split("\\")[2:])
except FileNotFoundError:
print(f"Error: File {input_path} not found.")
except Exception as e:
print(f"An error occurred while renaming the file: {str(e)}")
def speech_recognition_main():
# select training directory
train_dir = 'TrainingSet'
# select reference directory
ref_dir = 'ReferenceSet'
print('WELCOME TO UNIVERSAL TRANSLATOR. MAKE YOUR SELECTION:')
print('Enter 1 to train a new CNN')
print('Enter 2 to load an existing CNN')
user_entry = int(input('Enter your choice: '))
if user_entry == 1:
# Train a new CNN
print('Be patient until statistics are displayed...')
cnn_trained, audio_files = CNN_Train(train_dir)
input('CNN training is complete. Press enter to start using CNN.\n')
elif user_entry == 2:
# Load an existing CNN
cnn_file = input('Enter CNN name (with .h5 extension): ')
try:
# load the CNN and assign it to cnn_trained
cnn_trained = models.load_model(cnn_file)
audio_files = get_audio_files(train_dir)
except:
print('Enter a valid file name next time... Leaving... Bye')
return
else:
print('Enter a valid choice next time... Leaving... Bye')
return
# Now there is a trained CNN loaded (either a new one or an existing one).
# Use trained CNN to make predictions:
input('PRESS ENTER AND IMMEDIATELY SAY A WORD INTO YOUR MICROPHONE')
print('Recording...')
# Record audio
# setup recording parameters for microphone
sample_rate = 16000
duration = 2 # seconds
# create the object that performs recording:
rec_data = sd.rec(int(duration * sample_rate), samplerate=sample_rate, \
channels=1)
sd.wait()
# playback the voice recorded:
print('Replaying recording...')
139
140
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
9 CNNs for Speech Recognition
sd.play(rec_data, samplerate=sample_rate)
sd.wait()
# read languages from reference directory and store them into a list
languages = [d for d in os.listdir(ref_dir) \
if os.path.isdir(os.path.join(ref_dir, d))]
for idx, lang in enumerate(languages, 1):
print(f"Enter {idx} for {lang}")
# select translation output language
try:
user_input = int(input('What is your language choice: '))
output_lang = languages[user_input - 1]
print(f'You selected {output_lang} as the output language.')
except:
print('Your selection is out of bounds... Leaving... Bye')
return
# CNN predicts the translation
cnn_prediction = CNN_Predict(rec_data[:, 0], sample_rate, \
cnn_trained, audio_files)
# Get the reference file path
full_ref_path = os.path.join(ref_dir, output_lang, \
str(cnn_prediction).replace("TrainingSet\\English\\", ""))
full_ref_path = \
f"ReferenceSet\\{output_lang}\\" + rename_wav_file(full_ref_path)
# ads: audio data store is an object that holds audio files
try:
PlayBack_Translation(rec_data[:, 0], sample_rate, full_ref_path)
except:
print(f'translation for {output_lang} unavailable in {cnn_prediction}')
user_entry = input('Do you want to save CNN? (y/n) ')
if user_entry == 'y':
file_name = input('Enter file name (with .h5 extension): ')
cnn_trained.save(file_name)
print('Have a predictably nice day!')
speech_recognition_main()
A sample implementation of the CNN_Train function is given below. In lines 37 and 39, the
input audio files are converted to spectrograms. Their positions in the training folder are recorded
as the output values in line 42. Starting in line 49, the CNN given in Figure 9.2 is defined. The input
layer is defined in line 50; the first CL with 12 filters each of size (3 × 3) using padding so that the
output size will be the same as the input:
layers.Conv2D(12, (3x3), padding = 'same')
In lines 64 and 65, RELU and pooling layers are defined, respectively:
layers.Activation('relu')
layers.MaxPooling2D((2,2),padding = 'same')
The remaining CL–PL pairs and RELU units are defined between lines 67 and 84. The unrolling
layer is in line 86. A fully-connected ANN is defined in line 87:
layers.Dense(num_classes, activation = 'softmax')
where the outputs are generated as probabilities. Training the CNN is in line 96:
model.fit(XTrain. YTrain, epoch = 50, batch_size = 32, shuffle=True)
where the number of epochs is set to 50, and the option of reordering sample data at each epoch is
enabled.
9.4 Python Script for Speech Recognition Using a CNN
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
import numpy as np
from keras import layers, models
import os
from Speech_Spectrograms import speech_spectrograms
# CNN_Train Summary of this function goes here
# Train a new CNN using information from TrainingSet directory
# Then return trained CNN
# load the training audio dataset into ads_train.
# train_dir contains the path to the dataset.
# Function for searching mp3 files in a directory and its subfolders
def get_audio_files(directory):
mp3_files = []
for root, _, files in os.walk(directory):
for file in files:
if file.endswith('.wav'):
mp3_files.append(os.path.join(root, file))
return mp3_files
def CNN_Train(train_dir):
# Load audio files from train_dir
# audio_files = [f for f in os.listdir(train_dir) if os.path.isfile(f)]
audio_files = get_audio_files(train_dir)
# Compute their mel-spectrograms using the speech_spectrograms function
segment_duration = 2 # Adjust based on your data
frame_duration = 0.025
hop_duration = 0.010
num_bands = 40
epsil = 1e-6
# convert the audio samples into images by generating their
# spectrograms
X_train = speech_spectrograms(audio_files, segment_duration, \
frame_duration, hop_duration, num_bands)
X_train = np.log10(X_train + epsil)
# Assign labels based on the order of the files in the directory
Y_train = np.arange(len(audio_files))
# append a depth of 1 to specsize to get the input size of the CNN
input_shape = (num_bands, X_train.shape[2], 1)
num_classes = len(audio_files)
# Define the architecture of the CNN
model = models.Sequential([
layers.InputLayer(input_shape=input_shape),
# create the first convolution layer, it has 12 convolutional
# filters, each filter has a height and width of 3 and the input is
# zero padded.
# the input is padded with the appropriate number of zeros so that
# the size of the output is the same size as the input.
layers.Conv2D(12, (3, 3), padding='same'),
# normalize the output of the convolution between the range -1
# and 1. this prevents the updated weights from increasing
# explosively during training.
# layers.BatchNormalization(),
# apply rectilinear unit activation function (ReLU) that sets all
# negative values to zero.
layers.Activation('relu'),
layers.MaxPooling2D((2, 2), padding='same'),
layers.Conv2D(24, (3, 3), padding='same'),
layers.BatchNormalization(),
layers.Activation('relu'),
layers.MaxPooling2D((2, 2), padding='same'),
layers.Conv2D(48, (3, 3), padding='same'),
141
142
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
9 CNNs for Speech Recognition
layers.BatchNormalization(),
layers.Activation('relu'),
layers.MaxPooling2D((3, 3), padding='same', strides=(2, 2)),
layers.Conv2D(48, (3, 3), padding='same'),
layers.BatchNormalization(),
layers.Activation('relu'),
layers.Conv2D(48, (3, 3), padding='same'),
layers.BatchNormalization(),
layers.Activation('relu'),
layers.MaxPooling2D((1, 13)),
layers.Flatten(),
layers.Dense(num_classes, activation='softmax')
])
# Compile the model
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
# train the CNN
model.fit(X_train, Y_train, epochs=50, batch_size=32, shuffle=True)
return model, audio_files
A sample implementation of the CNN_Predict function that generates a prediction by the
trained CNN is presented below. The inputs of this function are the recorded user input, data
sample rate, trained CNN object and address of the TrainingSet folder. After setting up audio
parameters in lines 18–22, a spectrogram is computed for the user input audio in lines 26–29
and saved in the X_test object. Using the trained CNN, a prediction for this input is generated
in line 31:
predictions = model.predict(X_test)
where the object named predictions contains the probabilities for all CNN outputs. The index
for the output with the maximum probability is obtained in line 32:
predicted_class = np.argmax(predictions,axis=1)[0]
which is then used to select the audio files in the TrainingSet folder.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import os
import numpy as np
import soundfile as sf
from Speech_Spectrograms import speech_spectrograms
# CNN_Predict: Use CNN to predict the word
def CNN_Predict(rec_data, sample_rate, model, audio_files):
# Save audio data to a temporary file
temp_file = "temp.wav"
sf.write(temp_file, rec_data, sample_rate)
# Compute the mel-spectrogram from the audio data
# set audio parameters
# Duration of the entire recording
segment_duration = len(rec_data) / sample_rate
frame_duration = 0.025
hop_duration = 0.010
num_bands = 40
epsil = 1e-6
9.4 Python Script for Speech Recognition Using a CNN
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# compute speech spectrograms, they will be the inputs
# to the CNN
X_test = speech_spectrograms([temp_file], segment_duration, \
frame_duration, hop_duration, num_bands)
X_test = np.log10(X_test + epsil)
X_test = np.expand_dims(X_test, axis=-1)
# Use the model to predict the class
predictions = model.predict(X_test)
predicted_class = np.argmax(predictions, axis=1)[0]
# Delete the temporary file
os.remove(temp_file)
return audio_files[predicted_class]
A sample Python implementation of the speech_spectrograms function is given below,
which computes the spectrograms for audio files. Input parameters for this function include the
address of the folder where the audio files are stored and audio file characteristics such as the
segment and frame durations used in the audio recordings.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
import librosa
import numpy as np
def speech_spectrograms(files, segment_duration, frame_duration, \
hop_duration, num_bands, sr=16000):
print("Computing speech spectrograms...")
hop_length = int(hop_duration * sr)
all_spectrograms = []
# 1. Calculate the spectrograms for all files
for file in files:
y, _ = librosa.load(file, sr=sr)
mel_spec = librosa.feature.melspectrogram(y=y, sr=sr, n_fft=512,\
hop_length=hop_length, n_mels=num_bands)
all_spectrograms.append(mel_spec)
# 2. Determine the maximum width among all spectrograms
max_width = 419
final_spectrograms = []
# 3. Pad each spectrogram to the maximum width
for i, mel_spec in enumerate(all_spectrograms):
spec_width = mel_spec.shape[1]
left = (max_width - spec_width) // 2
padded_spec = np.zeros((num_bands, max_width))
# Diagnostic information
try:
padded_spec[:, left:left + spec_width] = mel_spec
except ValueError as e:
print(f"Error while processing file {files[i]}: {e}")
continue
final_spectrograms.append(padded_spec)
if (i + 1) % 1000 == 0:
print(f"Processed {i + 1} files out of {len(files)}")
print("...done")
return np.array(final_spectrograms)
143
144
9 CNNs for Speech Recognition
9.5 Matlab Script for Speech Recognition Using a CNN
A sample Matlab script is presented below, which implements a simple CNN with the architecture shown in Figure 9.2 constructed for speech recognition. This implementation uses a similar
structure to the Python script presented in Section 9.4 for ease of comparison for the reader. The
audio files for training and reference pronunciations are assumed to have been created and stored
in TrainingSet and ReferenceSet folders and their sub-folders as defined above.
In the Matlab script below, in lines 14–16, the user is prompted to either train a new CNN or load
a previously trained and saved one. If training a new CNN is chosen by entering 1, the function
named cnn_train is called in line 21:
cnn_trained = CNN_Train(train_dir);
where train_dir is a pointer containing the address of the TrainingSet folder. A new CNN is
created by CNN_ train, which is then saved in an object named cnn_trained. If the user wants
to use a previously trained CNN, this is loaded into cnn_trained in lines 30 and 31:
cnn_trained = load_data.cnn_trained;
After setting the recording parameters such as sampling and bit-per-sample rates and recording
duration in lines 47–50, the user is prompted to say a phrase into the microphone of the device,
which is recorded as an object named recObj in lines 51–53. This recording is first played back in
line 59, and, in lines 64–83, the user is asked to select a target language into which the phrase will
be translated. In line 85, CNN_predict is called:
cnn_prediction = CNN_Predict(recObj,cnn_trained);
where cnn_prediction contains the location of the prediction, which is then played back in
the user-selected language in line 90. Finally, in lines 96–100, the user is asked whether the trained
CNN is to be saved for later use.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
% The City College of New York, City University of New York
% Written by Ricardo Valdez
% August, 2020
% Speech Recognition and Translator System using CNNs
clc;
clear;
% select training directory
train_dir = 'TrainingSet';
% select reference directory
ref_dir = 'ReferenceSet';
message = sprintf('WELCOME TO UNIVERSAL TRANSLATOR. MAKE YOUR SELECTION:');
disp(message);
disp('Enter 1 to train a new CNN');
disp('Enter 2 to load an existing CNN');
user_entry = input('Enter your choice: ');
if (user_entry == 1)
% train a new CNN:
disp('Be patient until statistics are displayed...');
cnn_trained = CNN_Train(train_dir);
input('CNN training is complete. Press enter to start using CNN.\n');
elseif (user_entry == 2)
% use an existing CNN:
cnn_file = input('Enter CNN name (dont forget .mat extension): ', 's');
cnn_path = pwd;
disp(cnn_file);
try
% load the CNN and assign it to cnn_trained
load_data = load(fullfile(cnn_path,cnn_file));
9.5 Matlab Script for Speech Recognition Using a CNN
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
cnn_trained = load_data.cnn_trained;
catch
disp('Enter a valid file name next time... Leaving... Bye');
return;
end
else
disp('Enter a valid choice next time... Leaving... Bye');
return;
end
% Now there is a trained CNN loaded (either a new one or an existing one).
% Use trained CNN to make predictions:
input('PRESS ENTER AND IMMEDIATELY SAY A WORD INTO YOUR MICROPHONE');
disp('Recording...');
% setup recording parameters for microphone
sample_rate = 16000;
bits_per_sample = 8;
noof_channels = 1;
recording_duration = 2;
% create the object that performs recording:
recObj = audiorecorder(sample_rate, bits_per_sample, noof_channels);
% recording now:
recordblocking(recObj, recording_duration);
% playback the voice recorded:
disp('Replaying recording...');
play(recObj);
recording_duration = recObj.TotalSamples / recObj.SampleRate;
pause(recording_duration);
% read languages from reference directory and store them into a list
directories = dir(ref_dir);
languages = {};
for i = 1:length(directories)
directory = directories(i);
if directory.isdir && ∼strcmp(directory.name,...
'.') && ∼strcmp(directory.name,'..')
% append languages with directory.name
languages{length(languages)+1} = directory.name;
fprintf('Enter %d for %s\n', length(languages), directory.name);
end
end
try
% select translation output language
user_input = input('What is your language choice: ');
output_lang = languages{user_input};
fprintf('You selected %s as the output language.\n', output_lang);
catch
fprintf('Your selection is out of bounds... Leaving... Bye\n');
return;
end
% CNN predicts the translation
cnn_prediction = CNN_Predict(recObj, cnn_trained);
full_ref_path = fullfile(ref_dir, output_lang, char(cnn_prediction));
try
% ads: audio data store is an object that holds audio files
ref_ads = audioDatastore(full_ref_path);
PlayBack_Translation(recObj, ref_ads);
catch
fprintf('translation for %s is not available in %s\n',output_lang, ...
char(cnn_prediction));
end
user_entry = input('Do you want to save CNN? (y/n) ','s');
if(user_entry == 'y')
file_name = input('Enter file name (do not forget .mat extension): ',...
's');
save(file_name, 'cnn_trained');
else
end
disp('Have a predictably nice day!');
145
146
9 CNNs for Speech Recognition
A sample implementation of the CNN_Train function is presented below. Between lines 37
and 111, this function defines the architecture of the CNN defined in Figure 9.2 as follows. In
lines 40 and 47, the input layer and the first CL are defined, respectively. The first CL uses Fcl = 12
filters, each with a filter size of fcl = 3. Lines 51 and 54 specify the normalization and RELU units,
respectively. The first PL, using a pool size of p = 3 and stride of spl = 2, is defined in line 58. In
lines 63 to 69, the second CL (with Fcl = 24 and fcl = 3), RELU and the second PL (with p = 3 and
stride of spl = 2) are defined. Similarly, the third CL (with Fcl = 48 and fcl = 3), RELU and the third
PL (with p = 3 and stride of spl = 2) are specified between lines 74 and 80. The fourth and fifth
CLs (both with Fcl = 48 and fcl = 3) and the fourth PL (with p = 3 and stride of spl = 2) are defined
in lines 85, 92 and 98, respectively. An ANN whose outputs are the number of unique phrases to
be identified by the CNN is defined in line 103. To present the ANN outputs as probabilities as
opposed to exclusive binary values, a softmaxLayer is added to the CNN in line 107. Once these
layers are defined, the CNN is trained using the audio files saved in the TrainingSet directory:
net = trainNetwork(XTrain, YTrain, layers, options);
where textttXTrain and XTrain are the sample inputs and outputs for training, respectively,
layers is the CNN defined above and the options object is populated in lines 114 to 117
to use a gradient descent algorithm for optimization (i.e. adam), a learning rate of 0.0003,
maximum number of epochs as 25 and reordering input samples at each epoch (i.e. Shuffle
every-epoch). The trained CNN, saved in an object named net, is returned by the function.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
function net = CNN_Train(train_dir)
%CNN_Train Summary of this function goes here
% Train a new CNN using information from TrainingSet directory
% Then return trained CNN
% load the training audio dataset into ads_train.
% train_dir contains the path to the dataset.
adsTrain = audioDatastore(train_dir, ...
'IncludeSubfolders',true, ...
'LabelSource','foldernames');
% set audio parameters
fs = 16000;
segmentDuration = length(audioread(adsTrain.Files{1})) / fs;
frameDuration = 0.025;
hopDuration = 0.010;
numBands = 40;
epsil = 1e-6;
% convert the audio samples into images by generating their
% spectrograms
spectrograms = speechSpectrograms(adsTrain,segmentDuration, ...
frameDuration,hopDuration,numBands,fs);
% create input and expected output matrices to train on
XTrain = log10(spectrograms + epsil);
YTrain = adsTrain.Labels;
% get the dimensions of XTrain
sz = size(XTrain);
% set specSize to the height and width of a single spectrogram
specSize = sz(1:2);
% append a depth of 1 to specsize to get the input size of the CNN
inputSize = [specSize 1];
% get the total number of unique labels
numLabels = numel(categories(YTrain));
% Define the architecture of the CNN
layers = [
% set the size of the input image as the size of a single
% spectrogram
imageInputLayer(inputSize)
9.5 Matlab Script for Speech Recognition Using a CNN
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
% create the first convolution layer, it has 12 convolutional
% filters, each filter has a height and width of 3 and the input is
% zero padded.
% the input is padded with the appropriate number of zeros so that
% the size of the output is the same size as the input.
convolution2dLayer(3,12,'Padding','same')
% normalize the output of the convolution between the range -1
% and 1. this prevents the updated weights from increasing
% explosively during training.
batchNormalizationLayer
% apply rectilinear unit activation function (ReLU) that sets all
% negative values to zero.
reluLayer
% apply max pooling using a pooling size of (3x3), and using a
% stride of 2
maxPooling2dLayer(3,'Stride',2,'Padding','same')
% create a second convolution layer, it has 24 convolutional
% filters, each filter has a height and width of 3 and the input is
% zero padded.
convolution2dLayer(3,24,'Padding','same')
batchNormalizationLayer
reluLayer
% apply max pooling using a pooling size of (3x3), and using a
% stride of 2
maxPooling2dLayer(3,'Stride',2,'Padding','same')
% create a third convolution layer, it has 48 convolutional
% filters, each filter has a height and width of 3 and the input is
% zero padded.
convolution2dLayer(3,48,'Padding','same')
batchNormalizationLayer
reluLayer
% apply max pooling using a pooling size of (3x3), and using a
% stride of 2
maxPooling2dLayer(3,'Stride',2,'Padding','same')
% create a fourth convolution layer, it has 48 convolutional
% filters, each filter has a height and width of 3 and the input is
% zero padded.
convolution2dLayer(3,48,'Padding','same')
batchNormalizationLayer
reluLayer
% create a fifth convolution layer, it has 48 convolutional
% filters, each filter has a height and width of 3 and the input is
% zero padded.
convolution2dLayer(3,48,'Padding','same')
batchNormalizationLayer
reluLayer
% apply max pooling using a pooling size of (1x13), and using a
% stride of 1
maxPooling2dLayer([1 13])
% create the ouput layer
% the number of neurons in the output layer is equal to the number
% of unique labels
fullyConnectedLayer(numLabels)
% use a softmax layer to convert the outputs to a set of
% probalities where each output represents the probability that the
% image is a corresponding label.
softmaxLayer
% add a classification layer to make each output mutually
% exclusive, in other words there's only a single correct label for
% each image
classificationLayer];
% specify training parameters
options = trainingOptions('adam', ...
147
148
114
115
116
117
118
119
120
9 CNNs for Speech Recognition
'InitialLearnRate',3e-4, ...
'MaxEpochs',25, ...
'Shuffle','every-epoch');
%'Verbose',false);
% train the CNN
net = trainNetwork(XTrain,YTrain,layers,options);
end
We now present a sample implementation of a function named CNN_Predict, whose inputs
are the recorded audio input from the user and the trained CNN. After the audio input is converted
to a spectrogram in lines 19 to 21 (i.e. the object called XTest), it is given as an input to the CNN
in line 23, which returns the prediction as one of the outputs of the CNN:
[Prediction, ∼ ]=classify(net, XTest);
where net is the trained CNN. The output of classify is Prediction object, which contains
the prediction generated by the CNN.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
function [Prediction] = CNN_Predict(recObj, net)
%CNN_Predict: Use CNN to predict the word
y = getaudiodata(recObj);
Fs = recObj.SampleRate;
audiowrite('temp.wav',y,Fs);
adsTest = audioDatastore('temp.wav');
% set audio parameters
fs = 16000;
segmentDuration = length(audioread(adsTest.Files{1})) / fs;
frameDuration = 0.025;
hopDuration = 0.010;
numBands = 40;
epsil = 1e-6;
% compute speech spectrograms, they will be the inputs
% to the CNN
XTest = speechSpectrograms(adsTest,segmentDuration,frameDuration,...
hopDuration,numBands,fs);
XTest = log10(XTest + epsil);
[Prediction,∼] = classify(net, XTest);
% delete temporary file
if(isfile('temp.wav'))
delete temp.wav;
end
end
A sample implementation of the function named Playback_Translation is presented
below. This function replays the prediction found by the trained CNN. Inputs to this function are
the predicted object returned by the CNN_predict function and a pointer to the ReferenceSet
directory where the reference audio files are stored. After locating the reference audio file in line
21, it is played in line 22.
1
2
3
4
5
6
7
8
9
function [] = PlayBack_Translation(recObj, ref_ads)
%PlayBack_Translation
% enable text to speech
NET.addAssembly('System.Speech');
obj = System.Speech.Synthesis.SpeechSynthesizer;
% set volume of text to speech
obj.Volume = 100;
% text to speech
9.5 Matlab Script for Speech Recognition Using a CNN
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Speak(obj, 'Your word');
% playback the user's word
play(recObj);
recording_duration = recObj.TotalSamples / recObj.SampleRate;
pause(recording_duration);
% text to speech
Speak(obj, 'Translation');
% play the translated word from a reference file
[y,Fs] = audioread(ref_ads.Files{1});
sound(y,Fs);
end
A sample implementation for the speech_Spectrograms function is presented below. It
computes the spectrograms for a the audio file(s) that is (are) passed to it. Inputs to this function
include a reference to the folder where the audio file is stored and the audio characteristics of the
file, such as segment and frame durations.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
% speechSpectrograms(ads,segmentDuration,frameDuration,hopDuration,numBands)
% computes speech spectrograms for the files in the datastore ads.
% segmentDuration is the total duration of the speech clips (in seconds),
% frameDuration the duration of each spectrogram frame, hopDuration the
% time shift between each spectrogram frame, and numBands the number of
% frequency bands.
function X = speechSpectrograms(ads,segmentDuration,frameDuration,...
hopDuration,numBands,fs)
disp("Computing speech spectrograms...");
%fs
= 16e3;
FFTLength = 512;
persistent filterBank
if isempty(filterBank)
filterBank = designAuditoryFilterBank(fs,'FrequencyScale','mel',...
'FFTLength',FFTLength,...
'NumBands',numBands,...
'FrequencyRange',[50,7000]);
end
numHops = ceil((segmentDuration - frameDuration)/hopDuration);
numFiles = length(ads.Files);
X = zeros([numBands,numHops,1,numFiles],'single');
for i = 1:numFiles
x = read(ads);
frameLength = round(frameDuration*fs);
hopLength = round(hopDuration*fs);
[∼,∼,∼,spec] = spectrogram(x,hann(frameLength,'periodic'),...
frameLength - hopLength,FFTLength,'onesided');
spec = filterBank * spec;
% If the spectrogram is less wide than numHops, then put spectrogram in
% the middle of X.
w = size(spec,2);
left = floor((numHops-w)/2)+1;
ind = left:left+w-1;
X(:,ind,1,i) = spec;
if mod(i,1000) == 0
disp("Processed " + i + " files out of " + numFiles)
end
end
disp("...done");
end
149
150
9 CNNs for Speech Recognition
9.6 Concluding Remarks
CNN applications for speech recognition have great potential for significant impact on real-life
applications such as automatic subtitles, universal speech recognition and background noise cancellation. Speech recognition also enables a hands-free way for people to interact with intelligent
systems, making these systems more accessible, especially for disabled and elderly people.
In designing a CNN architecture for speech recognition tasks, spectrogram images must be
down-sampled sufficiently by convolutional and pooling layers before being fed to the ANN.
For different applications executing in various venues, designers should experiment with CNN
architectures, including the number of CL–PL pairs, filter quantities and sizes, and the number of
ANN layers and neurons in each layer to obtain an acceptable trade-off between training cost and
prediction accuracy.
9.7 Exercises for Chapter 9
1) Implement a CNN similar to the one presented in this chapter to recognize the following
spoken words in English and translate them into Japanese:
good morning
thank you
how much
airport
grocery store
too expensive taxi stand restaurant bus stop goodbye
2) Implement Exercise 1 for translation from Korean to German.
3) Implement Exercise 1 for translation from Japanese to Hindu.
4) For the example in Exercise 1, increase the number of CL–PL pairs in the CNN, and observe
any improvement in the prediction capability of the CNN.
5) For the example in Exercise 1, decrease the number of CL–PL pairs in the CNN, and observe
any impairment in prediction capability of the CNN.
6) Similar to Exercise 1, except that the input language is Polish and the target language is
Spanish.
7) Similar to Exercise 1, except that the input language is Korean and the target language is
Chinese.
8) Similar to Exercise 1, except that the input language is Filipino and the target language is
English.
9) Similar to Exercise 1, except that the input language is Swedish and the target language is
English.
10) Similar to Exercise 1, except that the input language is English and the target language is
German.
151
10
Recurrent Neural Networks
10.1 Introduction
There are many real-life systems whose outputs depend on the sequence in which the inputs are
applied to them. For example, in understanding natural languages, one has to consider multiple
words to make sense of a given sentence (e.g. subject, verb, prepositions, adverbs and adjectives), as
opposed to analysing each word independently. The next state of such a system depends not only
on the current input applied to it but also on the previous inputs. Such a recurrent system
possesses the ability to utilize the knowledge that it previously generated, be its previous outputs
or other relevant information. The ANNs that we studied in Chapters 2 to 6 consider each input
as being independent of the previously applied inputs and, therefore, are not suitable to handle a
sequence of inputs.
A recurrent neural network (RNN) is a type of artificial neural network, but with a
significant distinction that its intermediate state information and/or outputs are fed as inputs into
other neurons (hence the word recurrent in the name). Similar to other neural networks such
as ANNs or CNNs, RNNs are also trained by existing field data. However, while the inputs and
outputs of ANNs and CNNs are supposed to be independent for each sample, RNN outputs depend
on previously applied inputs, their outputs and some of the information created in the process
(e.g. so-called hidden states). In Figure 10.1, two versions of an RNN are shown: compact and
unfolded through time. As can be seen in the unfolded version, a previous hidden state of an RNN
generated by an earlier input is re-used at a later step when processing a new input. For example,
when predicting ŷ t at step t, the RNN uses both the inputs of xt and the hidden state information
generated at step (t − 1) for input xt−1 (i.e. ht−1 ). As step (t − 1) information influences the output
at step t, the hidden state generated at time t has an impact on prediction ŷ t+1 at step (t + 1).
Therefore, step (t − 1) indirectly influences step (t + 1), which is why they are called recurrent
networks. This re-use of internal information creates a type of internal memory, which makes
RNNs suitable for temporal applications, including language translation, speech recognition, and
image captioning. RNNs are essential building blocks of many popular AI applications, such as
Siri, voice search, Google Translate and others [86].
The main elements of an RNN can be listed as follows:
Input layer: the layer that accepts inputs (typically in the form of a sequence) at every step.
● Hidden Layer: the layer that stores information generated by current and previous steps; it represents the memory of an RNN and is updated at every step using current inputs and previous
hidden state information. It is also referred to as the encoder of an RNN.
●
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
152
10 Recurrent Neural Networks
Figure 10.1 Input, hidden and output layers of an RNN and a recurrent connection combining hidden
states (compact and unfolded-through-time presentations).
Recurrent Connection: the connection combining information from previous hidden layers
with the current input; this is the most important feature of an RNN in handling a sequence of
inputs.
● Output Layer: the layer that generates predictions at every step (or after multiple inputs are
applied depending on the RNN architecture as discussed later in this chapter). The output layer
is also referred to as the decoder of an RNN.
●
In the context of an RNN, the encoder is the part of the network that processes an input
sequence and encodes it into a fixed-length vector representation. In this way, an RNN can evaluate
input sequences in a numerical form. The output of the encoder is called the current hidden
state, which combines the current input with the previously generated hidden layer outputs (i.e.
previous hidden states). The purpose of the encoder is to ensure that previous inputs applied to
the RNN are taken into consideration while the current input is processed by the RNN. The output
of the encoder (i.e. the current hidden state) is then passed to the decoder of the RNN for further
processing.
In an RNN, the decoder is the component responsible for generating an output sequence based
on a fixed-length hidden-state vector produced by the encoder. Operation of the decoder is similar
to an encoder, but in reverse: it takes the hidden state as its initial input and generates an element
of the output.
There are different types of RNNs, where the outputs are generated either at every step, after
multiple inputs are applied, or any combination of the two cases [87]. A one-to-one RNN
generates one output for each input applied to it (e.g. a search engine suggesting the next word
based on the last word that you typed), whereas a one-to-many RNN produces multiple outputs
for a single input (e.g. in generating music). In many-to-one (e.g. in sentiment analysis) and
many-to-many RNNs (e.g. in natural language translation), a sequence of inputs generates a
single or a sequence of outputs, respectively.
In the remainder of this chapter, without loss of generality, we will focus on a one-to-one single
cell RNN for generating hidden states, outputs and partial derivations of loss, as well as updating
10.2 One-to-One Single RNN Cell
weight matrices during forward and back propagation operations. In Chapter 11, we present a
many-to-many RNN utilizing many cells for a simple chatbot example. The behaviour of other
types of RNNs follow the same principles of operation as one-to-one RNNs but employ matrices
with larger dimensions to represent their inputs, weights and outputs.
10.2 One-to-One Single RNN Cell
Let us now analyse the operation of a simple RNN, where there is one input (e.g. a single letter
selected from a given alphabet of letters), one hidden layer and one output (e.g. a single character
from the same alphabet). This type of an RNN is referred to as a one-to-one RNN. Figure 10.2a
presents a compact depiction of a one-to-one single RNN cell. This compressed RNN cell functions
as a series of RNN cells concatenated one after another, as shown in Figure 10.2b. The hidden
state generated at the immediate previous step (i.e. ht−1 ) is fed into the RNN encoder at step t. The
encoder converts the input into a vector so that the RNN can process it and by incorporating the
previous hidden states, generates the current hidden state, based on which the decoder generates
the RNN output. Let us now examine the elements of an RNN, as shown in Figure 10.2:
Xt : input vector at step t
U: weight matrix scaling the input values
● b: bias vector for encoder
● h : hidden state at step t
t
● W: weight matrix controlling the previous hidden state information of h
t−1
● tanh unit: an example activation function used in encoder to filter out weaker signals
● V: weight matrix for controlling current hidden state information
● c: bias vector for decoder
● Ω : raw output vector generated by encoder before being fed into the softmax unit at
t
step t
● softmax unit: a unit to convert decoder outputs to probabilities
̂ t : output vector containing the prediction generated by the RNN at step t
● Y
●
●
An encoder of an RNN cell applies the weight matrix of U to the current input vector Xt while
scaling the previous hidden state ht−1 by weight matrix W and scaling by a bias vector b, and
then sums them and feeds the result into the activation unit (e.g. tanh) to generate the current
hidden state information, ht (Figure 10.2b). A decoder, on the other hand, scales the hidden
state information ht by a weight matrix of V and a bias vector of c to generate raw outputs of
Ωt , which are then fed into an activation unit (e.g. softmax) to obtain an output prediction of
Ŷ t . It is important to note that, in Figure 10.2b, during different steps, the same weight matrices
(i.e. U, b, W, V and c) are used for different inputs.
In an RNN, a recurrence formula is applied to an input at step t as follows:
ht = Θh (U ⋅ xt + W ⋅ ht−1 + b),
where ht and ht−1 are the current and previous hidden states, respectively, Θh is an activation function and xt is the current input. For most RNNs, the activation function Θh in the encoder is typically
chosen as tanh. It is preferred to other functions because it maintains non-zero values for longer
ranges of epochs during training. The pre-softmax outputs generated by the decoder are called
raw outputs and defined as the Ωt vector at step t:
Ωt = V ⋅ ht + c.
153
154
10 Recurrent Neural Networks
(a)
(b)
Figure 10.2 A one-to-one single RNN cell: (a) compressed, and (b) unfolded versions (note that the same
weight matrices, namely U, W, b, V and c, are used throughout the RNN over the multiple applications of
input sequences).
10.2 One-to-One Single RNN Cell
For certain applications, the outputs of the Ωt vector are required to be generated as probabilities.
However, it is possible that they may have negative values. To prevent such results, a unit called
softmax is employed, which converts the raw output values into probabilities with values between
0 and 1, inclusive, and hence eliminates negative values in outputs. With softmax, the sum of all
outputs add up to 1, where each output represents how confident the RNN is that it is the correct
prediction. For an input alphabet size of ||, output ŷ t is the value of the prediction at step t, which
is calculated as a probability based on the raw output of value at step t (i.e. Ωt ). Therefore, the
predicted output of ŷ t is computed as follows:
Ωt,1
e
⎡ ∑||
⎤
Ω
⎢ j=1 e t,j ⎥
⎢ eΩt,2 ⎥
⎢ ∑|| Ωt,j ⎥
eΩt
ŷ t = Θŷ = softmax(Ωt ) = ⎢ j=1 e ⎥ = ∑||
,
eΩt,j
⎢ ⋮ ⎥
j=1
⎢ Ωt,|| ⎥
⎢ ∑e|| Ωt,j ⎥
⎣ j=1 e ⎦
(10.1)
where || is the size of the alphabet , and Ωt,i is the ith element of the Ωt matrix. We can see in
Eqn. (10.1) that the ŷ t elements have values between 0 and 1 (excluding 0). It shows that elements
of raw output Ωt , whose values may be negative, are replaced with their power of e (i.e. eΩt,i ) and
divided by the summation of all raw output values as powers of e. The values for the ŷ t vector,
therefore, are probabilities with values between 0 and 1 (excluding 0).
Let us now consider the dimensions of the input, output and weight matrices in a one-to-one
single RNN cell for an input alphabet of four characters encoded with one-hot encoding (a special
encoding to represent characters as inputs for an RNN as discussed in Section 10.2.1), as shown in
Figure 10.3. For this example, in the encoder, the dimensions of input vector X and the U matrix
Figure 10.3 Input, weight and output vector
dimensions in a one-to-one single RNN cell for
an input alphabet of four characters encoded
with one-hot encoding.
155
156
10 Recurrent Neural Networks
are (4 × 1) and (1 × 4), respectively. Dimensions for each of ht−1 , W and b will be (1 × 1). Therefore,
the dimensions for the inputs applied to and the outputs generated by the activation unit (which
is realized as tanh in Figure 10.3) will be (1 × 1). In the decoder unit, the dimensions for both the
V and c matrices are (4 × 1). The activation function of the decoder (i.e. Θŷ ), realized as softmax
in Figure 10.3, generates an output prediction vector of Ŷ with the same dimensions as its inputs,
which are (4 × 1).
10.2.1 A Simple Alphabet and One-Hot Encoding
RNN algorithms cannot easily operate on textual data since numeric representation of text
is needed for algorithmic procedures. Text inputs for an RNN have to be pre-processed using
encoding techniques. One such method is called one-hot encoding [88, 89], where a
sparse vector is populated for each string character in the alphabet such that only a single element
of this vector is set to one, while the others are zero.
Let us now consider a simple alphabet  consisting of four letters, defined as  = {h, e, l, o}.
Using one-hot encoding, each character di in  is represented as a vector of length || (i.e. the total
number characters in ) with the character’s position in  set to one and remaining elements to
zeros (i.e. only one position of the vector is hot, whereas the remaining positions are cold, hence
the name one-hot encoding):
{
1 if i = di ,
o
(10.2)
di =
0 otherwise.
where i means the ith position in alphabet . For example, for d = h in alphabet  = {h, e, l, o},
the one-hot encoding is ho = [1, 0, 0, 0]† , where † represents the transpose operator. Similarly, for
d = e, the one-hot encoding is eo = [0, 1, 0, 0]† .
Note that one-hot encoding generates sparse vectors of length ||, of which only a single element
is one. This encoding can be prohibitively inefficient if an RNN uses the words in natural language
alphabets with tens of thousands of words as its inputs, which is the case for machine translation
problems. For these types of applications, different methods of encoding are preferred depending
on the input characteristics, including Word2Vec [90], GloVe [91] and sparse encoding [92].
Another popular method to reduce the dimensions of input vectors is to employ so-called
embedding layers, where the dimensions of input vectors are significantly reduced via weight
matrices of the embedding layer; we will look at this technique in Section 10.5 of this chapter.
For this simple example, where an alphabet with four letters is used, the expected output for
inputs, together with the previous input, are given in Table 10.1. This example assumes that,
as a user enters single inputs, the predictions by the RNN will lead the user to spell the word
Table 10.1 Previous and current inputs and expected outputs
for the simplistic example for a one-to-one RNN.
Previous
input
Current
input
Expected
ouput
-
h
e
h
e
l
e
l
l
l
l
o
10.2 One-to-One Single RNN Cell
hello. Therefore, if the user enters h, the RNN should generate e. After input h, if the user enters
e, the RNN should generate l. Similarly, two consecutive inputs of l should generate l and o,
respectively. Note that this simple example for operation of an RNN is designed to highlight the
use of previous hidden state information in generating the current output. The first time input l
is applied, the RNN is expected to generate an output of l, whereas the second l is expected to
generate an output of o. This shows the impact of previous input values in the current output. The
RNN’s ability to maintain a memory is what distinguishes it from ANNs and CNNs.
10.2.2 Forward and Back Propagation
It is time to demonstrate the operations when an input is applied to the one-to-one single cell
RNN given in Figures 10.2a and b using our simple example alphabet of  = {h, e, l, o}. Suppose a
sequence of input characters x1 , x2 , · · · is applied to the RNN. For a user input xt at time t, the RNN
computes a hidden state as follows:
ht = tanh(U ⋅ xt + W ⋅ ht−1 + b).
(10.3)
As shown in Figures 10.2a and b, the raw output before the softmax unit, Ωt , is computed as
Ωt = V ⋅ ht + c.
(10.4)
Recall from Eqn. (10.1) that the softmax unit converts elements of Ωt to probability values
between 0 and 1 and assigns these to ŷ t :
eΩt
,
ŷ t = ∑||
eΩt,j
j=1
(10.5)
where Ωt,j represents the jth element of the Ωt vector. Cross-entropy loss L [93] is an entity
defined to represent the difference between the RNN prediction and the real output value. For the
prediction ŷ t at step t, L is computed as
Lt = −yot ln(̂yt ),
(10.6)
where yot is the one-hot encoded value of the target (i.e. correctly predicted) output. For an input
sequence of length  , the total loss after  steps becomes
L=

∑
Lt .
(10.7)
t=1
In order to minimize the loss L in Eq. (10.7), its gradients with respect to RNN parameters such as
U, W, V, b and c must be computed. Computation of gradients corresponds to a back-propagation
operation starting from the predicted output back to the input, each step performed by a partial
derivation. Step-by-step derivations of these gradients are presented in Appendix D. We strongly
encourage the reader to go over this material to understand the actions taken to train an RNN. The
partial derivatives of L with respect to the weights are as follows:
𝜕Lt
= (̂yt − yot )
𝜕Ωt
(10.8)

𝜕L ∑
=
(̂yt − yot ) ⋅ h†t
𝜕V
t=1
(10.9)

𝜕L ∑
=
(̂yt − yot )
𝜕c
t=1
(10.10)
157
158
10 Recurrent Neural Networks
)
( t
( t−1
)

∑
∑
∏
𝜕L
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) hk−1
𝜕W
t=1
k=1
j=k
( t
( t−1
)
)

∏
𝜕L ∑ ∑
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) xk
𝜕U
t=1
k=1
j=k
)
( t
t−1 (

)
∏
𝜕L ∑ ∑
o
†
⊤
2
2
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) ,
=
𝜕b
t=1
k=1
j=k
(10.11)
(10.12)
(10.13)
where † represents the matrix transpose operation. Using these gradients, values of the U, W, V, b
and c matrices are updated after a back propagation by subtracting the current value scaled by a
learning rate of 𝜂 (similar to the learning rate 𝛼 used in ANNs in Chapter 2):
𝜕L
𝜕V
(10.14)
𝜕L
𝜕c
(10.15)
Vnew = V − 𝜂
cnew = c − 𝜂
𝜕L
𝜕W
(10.16)
𝜕L
𝜕U
(10.17)
Wnew = W − 𝜂
Unew = U − 𝜂
𝜕L
,
(10.18)
𝜕b
where subscript new is used to represent the updated parameter values. Since these parameters
are updated by using gradients of loss L, it is expected that the next forward propagation using
the updated parameters will yield a smaller loss. This is demonstrated in the numerical example
presented below.
bnew = b − 𝜂
10.3 A Numerical Example
Recall the simple alphabet of four characters:  = {h, e, l, o}. One-hot encodings of each
character in  are
ho ∶ [1, 0, 0, 0]†
eo ∶ [0, 1, 0, 0]†
lo ∶ [0, 0, 1, 0]†
oo ∶ [0, 0, 0, 1]† ,
where † represents the matrix transpose operation. Suppose the initial values for U, W, V, b and c
are randomly assigned as follows:
[
]
[
]
[ ]
U = 0.144 −0.232 −0.669 1.159
W = −0.587
b= 0
[
]†
[
]†
c= 0 0 0 0 .
V = −0.225 − 0.305 0.491 0.277
Let us now apply the sequence of inputs h, e, l and l, with the expectation that the RNN will
generate predictions of e, l, l and o, respectively, as presented in Table 10.1. Since we expect our
simple RNN to generate a single character output at a time as part of completing the word hello,
when the user applies the input character x1 = h, the RNN is expected to generate e. One-hot
encoding for this input is
x1o = [1, 0, 0, 0]† .
10.3 A Numerical Example
For input x1 = h, using Eq. (10.3), hidden state h1 is computed as follows:
h1 = Θh (U ⋅ x1 + W ⋅ h0 + b)
⎞
⎡1⎤
⎛
⎟
⎜[
] ⎢0⎥
= tanh ⎜ 0.144 −0.232 −0.669 1.159 ⋅ ⎢ ⎥ + [−0.587] ⋅ [0] + [0]⎟
⎟
⎢0⎥
⎜
⎟
⎢0⎥
⎜
⎣ ⎦
⎠
⎝
= [0.143].
From Eq. (10.4), we obtain raw output values as follows:
⎡−0.225⎤
⎡−0.032⎤
⎢
⎥
⎢
⎥
−0.305
⎥ ⋅ [0.1432] + [0] = ⎢−0.044⎥ .
Ω1 = V ⋅ h1 + c = ⎢
⎢ 0.491⎥
⎢ 0.070⎥
⎢ 0.277⎥
⎢ 0.040⎥
⎣
⎦
⎣
⎦
Applying the softmax function given in Eqn. (10.1), we have
⎡0.240⎤
⎢
⎥
0.237⎥
.
ŷ 1 = Θŷ (Ω1 ) = ⎢
⎢𝟎.𝟐𝟔𝟔⎥
⎢0.250⎥
⎣
⎦
Based on the highest value of the softmax output, for input x1 = h, the RNN output prediction
is ŷ 1 = l. However, from Table 10.1, the RNN should have predicted the output as yo1 = e. Using
Eqn. (10.6), the loss at Step 1 is computed as
⎡0⎤ ⎡0.240⎤
⎢ ⎥ ⎢
⎥
1
0.237⎥
L1 = −yo1 ln(̂y1 ) = − ⎢ ⎥ ln ⎢
= 1.440.
⎢0⎥ ⎢0.266⎥
⎢0⎥ ⎢0.258⎥
⎣ ⎦ ⎣
⎦
Continuing with the example, at step t = 2, the user applies input x2 = e with the expectation
that the RNN will generate output l (as given Table 10.1). One-hot encoding for input e is
x2o = [0, 1, 0, 0]† .
Let us compute the hidden state h2 as follows:
h2 = Θh (U ⋅ x2 + W ⋅ h1 + b)
⎞
⎡0⎤
⎛
⎟
⎜[
] ⎢1⎥
= tanh ⎜ 0.144 −0.232 −0.669 1.1585 ⋅ ⎢ ⎥ + [−0.587] ⋅ [0.143] + [0]⎟
⎟
⎢0⎥
⎜
⎟
⎢0⎥
⎜
⎣ ⎦
⎠
⎝
= −0.306.
The raw output values are computed as
⎡ 0.069⎤
⎡−0.225⎤
⎢
⎥
⎢
⎥
−0.305
⎥ ⋅ [−0.306] + [0] = ⎢ 0.093⎥ .
Ω2 = V ⋅ h2 + c = ⎢
⎢−0.150⎥
⎢ 0.491⎥
⎢−0.085⎥
⎢ 0.277⎥
⎣
⎦
⎣
⎦
159
160
10 Recurrent Neural Networks
The outputs after the softmax unit are
⎡0.271⎤
⎥
⎢
𝟎.𝟐𝟕𝟖⎥
.
ŷ 2 = Θŷ (Ω2 ) = ⎢
⎢0.218⎥
⎢0.233⎥
⎦
⎣
Based on the highest value of the softmax unit output, the RNN prediction is ŷ 2 = e, although
the expected prediction is yo2 = l as in Table 10.1. The loss at t = 2 is
⎡0⎤ ⎡0.271⎤
⎢ ⎥ ⎢
⎥
0
0.278⎥
= 1.523.
L2 = −yo2 ln(̂y2 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.218⎥
⎢0⎥ ⎢0.233⎥
⎣ ⎦ ⎣
⎦
We continue at step t = 3 by applying input x3 = l to the RNN with the expectation that output
l will be generated. One-hot encoding for l is
x3o = [0, 0, 1, 0]† .
Let us now compute hidden state h3 as follows:
h3 = Θh (U ⋅ x3 + W ⋅ h2 + b)
⎞
⎛
⎡0⎤
⎟
⎜[
] ⎢0⎥
= tanh ⎜ 0.144 − 0.232 − 0.670 1.159 ⋅ ⎢ ⎥ + [−0.587] ⋅ [−0.306] + [0]⎟
⎟
⎜
⎢1⎥
⎟
⎜
⎢0⎥
⎣ ⎦
⎠
⎝
= −0.454.
The raw outputs at step t = 3 are
⎡ 0.102⎤
⎡−0.225⎤
⎢
⎥
⎢
⎥
0.139⎥
−0.305⎥
Ω3 = V ⋅ h3 + c = ⎢
⋅ [−0.454] + [0] = ⎢
⎢−0.223⎥
⎢ 0.491⎥
⎢−0.126⎥
⎢ 0.277⎥
⎣
⎦
⎣
⎦
⎡0.281⎤
⎢
⎥
𝟎.𝟐𝟗𝟐⎥
ŷ 3 = Θŷ (Ω3 ) = ⎢
.
⎢0.203⎥
⎢0.224⎥
⎣
⎦
The highest value of the softmax function output indicates that the RNN-predicted output for
input x3 = l is ŷ 3 = e, which is as expected. At step t = 3, the loss is
⎡0⎤ ⎡0.281⎤
⎥
⎢ ⎥ ⎢
0.292⎥
0
= 1.593.
L3 = −yo3 ln(̂y3 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.203⎥
⎢0⎥ ⎢0.224⎥
⎦
⎣ ⎦ ⎣
At step t = 4, we apply input x4 = l to the RNN with the expectation that the prediction will be
o. One-hot encoding for l is
x4o = [0, 0, 1, 0]† .
10.3 A Numerical Example
Let us now compute the hidden state h4 as follows:
h4 = Θh (U ⋅ x4 + W ⋅ h3 + b)
⎛
⎡0⎤
⎞
⎜[
⎟
] ⎢0⎥
= tanh ⎜ 0.144 − 0.232 − 0.669 1.159 × ⎢ ⎥ + [−0.587] ⋅ [−0.454] + [0]⎟
⎜
⎢1⎥
⎟
⎜
⎢0⎥
⎟
⎣ ⎦
⎝
⎠
= [−0.382].
The raw output values are
⎡ 0.089⎤
⎡−0.225⎤
⎥
⎢
⎥
⎢
0.117⎥
−0.305⎥
⋅ [−0.382] + [0] = ⎢
Ω4 = V ⋅ h4 + c = ⎢
⎢−0.187⎥
⎢ 0.491⎥
⎢−0.106⎥
⎢ 0.277⎥
⎦
⎣
⎦
⎣
⎡0.276⎤
⎢
⎥
𝟎.𝟐𝟖𝟓⎥
ŷ 4 = Θŷ (Ω4 ) = ⎢
.
⎢0.210⎥
⎢0.228⎥
⎣
⎦
Based on the output from the softmax unit, the RNN predicts the output as ŷ 4 = e for input
x4 = l, whereas the expected output target is yo4 = o. For step t = 4, the loss is
⎡0⎤ ⎡0.276⎤
⎢ ⎥ ⎢
⎥
0
0.285⎥
= 1.478.
L4 = −yo4 ln(̂y4 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.210⎥
⎢1⎥ ⎢0.228⎥
⎣ ⎦ ⎣
⎦
As given in Eq. (10.7), the total loss after four steps is computed as
L=
4
∑
Lt = 1.440 + 1.523 + 1.593 + 1.478 = 6.034.
(10.19)
t=1
After completing the forward propagation for four steps, let us now consider back propagation
operations. The goal of back propagation is to compute the gradients for the loss function with
respect to the RNN parameters such that L value shown in Eqn. (10.19) is reduced after the weights
are updated. Once a back propagation is complete, a new forward propagation is performed and a
new loss function is computed, and so on. Similarly to training ANNs and CNNs, completion of one
forward and one back propagation step is often called one epoch. In realistic RNNs, thousands of
epochs are typically needed to obtain acceptable levels of loss. From Eq. (10.8), the gradient of L
with respect to predicted output ŷ is as follows:
⎡0.240⎤ ⎡0⎤ ⎡ 0.240⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L1
0.237⎥ ⎢1⎥ ⎢−0.763⎥
=
−
= (̂y1 − yo1 ) = ⎢
⎢0.266⎥ ⎢0⎥ ⎢ 0.266⎥
𝜕Ω1
⎢0.258⎥ ⎢0⎥ ⎢ 0.258⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
⎡0.271⎤ ⎡0⎤ ⎡ 0.271⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L2
0.278⎥ ⎢0⎥ ⎢ 0.278⎥
=
−
= (̂y2 − yo2 ) = ⎢
⎢0.218⎥ ⎢1⎥ ⎢−0.782⎥
𝜕Ω2
⎢0.233⎥ ⎢0⎥ ⎢ 0.233⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
161
162
10 Recurrent Neural Networks
⎡0.281⎤ ⎡0⎤ ⎡ 0.281⎤
⎢
⎥ ⎢ ⎥ ⎢
⎥
𝜕L3
0.292⎥ ⎢0⎥ ⎢ 0.292⎥
= (̂y3 − yo3 ) = ⎢
−
=
⎢0.203⎥ ⎢1⎥ ⎢−0.797⎥
𝜕Ω3
⎢0.224⎥ ⎢0⎥ ⎢ 0.224⎥
⎣
⎦ ⎣ ⎦ ⎣
⎦
⎡0.276⎤ ⎡0⎤ ⎡ 0.276⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L4
0.285⎥ ⎢0⎥ ⎢ 0.285⎥
.
=
−
= (̂y4 − yo4 ) = ⎢
⎢0.210⎥ ⎢0⎥ ⎢ 0.210⎥
𝜕Ω4
⎢0.228⎥ ⎢1⎥ ⎢−0.772⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
Using Eqs. (10.9), (10.10), (10.11), (10.12) and (10.13), we compute the gradients with respect to
V, c, W, U and b, respectively, as follows:
⎡−0.282⎤

⎢
⎥
∑
−0.436⎥
𝜕L
=
=
(̂yt − yot ) ⋅ h†t = ⎢
⎢ 0.558⎥
𝜕V
𝜕V
t=1
t=1
⎢ 0.159⎥
⎣
⎦

∑
𝜕Lt
⎡ 1.069⎤

⎢
⎥
∑
0.092⎥
𝜕L
=
=
(̂yt − yot ) = ⎢
⎢−1.103⎥
𝜕c
𝜕c
t=1
t=1
⎢−0.058⎥
⎣
⎦
(
)
t−1 (

t
)
∑
∑
∏
𝜕L
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) hk−1
𝜕W
t=1
k=1
j=k
[
]
= 0.147

∑
𝜕Lt
( t
)
t−1 (

)
∏
𝜕L ∑ ∑
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) xk
𝜕U
t=1
k=1
j=k
[
]
= 0.530 −0.273 −0.500 0.0
)
( t
t−1 (

)
∏
𝜕L ∑ ∑
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk )
𝜕b
t=1
k=1
j=k
[
]
= −0.2436 .
From Eqs. (10.14), (10.15), (10.16), (10.17) and (10.18), for a learning rate of 𝜂 = 0.1 (arbitrarily
chosen value), we now update V, c, W, U and b, respectively, as follows:
⎡−0.2818⎤ ⎡−0.196⎤
⎡−0.225⎤
⎥
⎢
⎥ ⎢
⎢
⎥
−0.436 ⎥ ⎢−0.262⎥
𝜕L ⎢−0.305⎥
=
− 0.1 ⎢
=
Vnew = V − 𝜂
⎢ 0.558 ⎥ ⎢ 0.435⎥
𝜕V ⎢ 0.491⎥
⎢ 0.159 ⎥ ⎢ 0.261⎥
⎢ 0.277⎥
⎦
⎣
⎦ ⎣
⎣
⎦
⎡ 1.069⎤ ⎡−0.107⎤
⎡0⎤
⎥
⎢
⎥ ⎢
⎢ ⎥
0.092⎥ ⎢−0.009⎥
0
𝜕L ⎢ ⎥
cnew = c − 𝜂
=
− 0.1 ⎢
=
⎢−1.103⎥ ⎢ 0.110⎥
𝜕c ⎢0⎥
⎢−0.058⎥ ⎢ 0.006⎥
⎢0⎥
⎦
⎣
⎦ ⎣
⎣ ⎦
10.4 Multiple Hidden Layers
Wnew = W − 𝜂
[
]
[
] [
]
𝜕L
= −0.587 − 0.1 0.147 = −0.602
𝜕W
𝜕L
Unew = U − 𝜂
𝜕U
[
]
[
]
= 0.144 −0.232 −0.669 1.159 − 0.1 0.530 −0.273 −0.500 0.0
[
]
= 0.091 −0.204 −0.619 1.159
[
] [
]
𝜕L [ ]
= 0 − 0.1 −0.2436 = 0.0244 ,
𝜕b
where the subscript new on the left-hand side of the equations represents the updated value of a
parameter. Using these new values, we now perform a new forward propagation, which yields the
following loss values (computations are not shown):
bnew = b − 𝜂
⎡0⎤ ⎡0.217⎤
⎢ ⎥ ⎢
⎥
1
0.237⎥
= 1.438
L1 = −yo1 ln(̂y1 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.290⎥
⎢0⎥ ⎢0.256⎥
⎣ ⎦ ⎣
⎦
⎡0⎤ ⎡0.239⎤
⎢ ⎥ ⎢
⎥
0
0.268⎥
L2 = −yo2 ln(̂y2 ) = − ⎢ ⎥ ln ⎢
= 1.369
⎢1⎥ ⎢0.254⎥
⎢0⎥ ⎢0.239⎥
⎣ ⎦ ⎣
⎦
⎡0⎤ ⎡0.249⎤
⎥
⎢ ⎥ ⎢
0.283⎥
0
= 1.437
L3 = −yo3 ln(̂y3 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.238⎥
⎢0⎥ ⎢0.230⎥
⎦
⎣ ⎦ ⎣
⎡0⎤ ⎡0.244⎤
⎥
⎢ ⎥ ⎢
0.275⎥
0
L4 = −yo4 ln(̂y4 ) = − ⎢ ⎥ ln ⎢
= 1.449
⎢0⎥ ⎢0.246⎥
⎢1⎥ ⎢0.235⎥
⎦
⎣ ⎦ ⎣
L=

∑
t=1
Lt = 1.438 + 1.369 + 1.437 + 1.449 = 5.692,
where the predictions are shown as boldface characters at each step. For the inputs of x1 = h,
x2 = e, x3 = l and x4 = l, the model in the second forward propagation generates the predictions
of ŷ 1 = l, ŷ 2 = e, ŷ 3 = e and ŷ 4 = e, respectively. Also, we observe that L is reduced to 5.692 from
its value of 6.034 at the first forward propagation. Although we do not present the results here due
to space constraints, using these updated values in a third forward propagation is now expected to
generate better predictions with smaller error values. If we repeat the training process for hundreds
of epochs, we will observe that the RNN predictions approach acceptable ranges. For this example,
we obtained the values for total loss as 2.060 and 1.346 for 100 and 1000 epochs, respectively.
10.4 Multiple Hidden Layers
One way of improving the prediction power of an RNN is to employ multiple hidden layers
(Fig. 10.4). In an RNN with additional hidden layers, information captured by the hidden states
163
164
10 Recurrent Neural Networks
Figure 10.4 Basic parts of a one-to-one RNN with multiple hidden layers (compact and unfolded
presentations).
Figure 10.5
A one-to-one RNN cell with two hidden layers.
for the same input vector is increased. The hidden layers operate in parallel to generate different
approximations at each step, of which the better performing ones will eventually yield more
accurate predictions during the training process. In Figure 10.5, we show a one-to-one RNN with
two hidden layers (also referred to as two cells). In this case, there are two encoder cells, which
are operating in parallel using different weights. We may envision that using two hidden layers in
an RNN as equivalent to using two neurons in an ANN layer. As discussed in Chapter 2, multiple
neurons in an ANN represent parallel and simultaneous searches for the same input values.
10.5 Embedding Layer
Figure 10.6 Vector dimensions in the one-input to one-output two-cell RNN shown in Figure 10.5, which
is equivalent to having more neurons in an ANN layer, as discussed in Chapter 2.
When multiple hidden layers are implemented in an RNN, the dimensions of the weight matrices
are accordingly increased to accommodate the additional weights due to the extra hidden layers. In
Figure 10.6 we show the dimensions of weight matrices for the RNN with two hidden layers given
in Figure 10.5. To better visualize the search power added by the extra hidden layers, the reader
should compare the weight matrices for this RNN with the ones given in Figure 10.3 employing a
single hidden layer. In the general case, by including  hidden layers, the matrix dimensions in a
one-to-one RNN are shaped as follows:
Encoder: X ∶ (|| × 1), U ∶ ( × ||), W ∶ ( ×  ), b ∶ ( × 1), H ∶ ( × 1)
Decoder: V ∶ (|| ×  ), c ∶ (|| × 1), Ω ∶ (|| × 1)
̂ ∶ (|| × 1)
● Output: Y
●
●
where || is the alphabet size and  is the number of hidden layers. Note that for the example
shown in Figure 10.6, we have || = 4 and  = 2.
10.5 Embedding Layer
Earlier in this chapter, we introduced one-hot encoding as a method for converting input characters
to numerical representations in an RNN implementation. However, there are serious drawbacks in
165
166
10 Recurrent Neural Networks
using one-hot encoding and methods similar to it in terms of the size of the input vectors. In one-hot
encoding, for each input character xto , the input vector size is (|| × 1) for an alphabet size of ||,
in which only a single element is set to one (i.e. one element is hot) and the remaining elements
are all zeros. Recall that, in the example presented in Section 10.2.1, we defined one-hot encoding
for input h as ho = [1, 0, 0, 0]† , where † represents the matrix transpose operation. However, if we
were to use a realistic dictionary with tens of thousands of elements, it is clear that one-hot encoding would have generated infeasibly large input vectors. For these situations, to reduce the input
vector size, we utilize embedding layers [94] at the front end of the RNN. The main goal of an
embedding layer is to convert sparse input vectors (e.g. one-hot encoded words or characters) into
dense vectors through matrix manipulations so that the computational complexity of RNN training
is significantly reduced. For an embedding layer, we define a weight matrix, called E, with dimensions of (e × ||), where e and || are the embedding width (sometimes referred to as embedding
size) and the alphabet size, respectively. A one-hot-encoded input element xto is multiplied by the
embedding matrix E before being applied to the RNN encoder as xt = E ⋅ xto . We can see from this
equation that, for an input xto with dimensions of (|| × 1), multiplication by E will result in an
input vector xt whose dimensions are reduced to (e × 1). As an example, suppose we have xto and
the elements of E given as follows:
⎡1⎤
⎢ ⎥
0
xto = ho = ⎢ ⎥ ,
⎢0⎥
⎢0⎥
⎣ ⎦
[
]
0.8 −0.3 0.1 −0.1
.
0.4 −0.4 0.1 −0.5
Implementing an embedding layer reduces the xto dimensions when applied to the RNN encoder
as follows:
⎡1⎤
[
] ⎢ ⎥ [ ]
0.8
−0.3
0.1
−0.1
0
0.8
⋅⎢ ⎥=
.
xt = E ⋅ xto =
0.4 −0.4 0.1 −0.5 ⎢0⎥
0.4
⎢0⎥
⎣ ⎦
Figure 10.7 presents a one-to-one single-cell RNN with an embedding layer of size 2. Comparing
the dimensions of the weights in Figure 10.7 with the example RNN without an embedding layer
in Figure 10.6, we can clearly see the reduced dimensions in weight matrices due to embedding.
In the general case, using an alphabet size of || and  hidden layers in a one-to-one RNN with
an embedding layer size of || × e, parameter matrix dimensions are shaped as follows:
Embedding: X o ∶ (|| × 1), E ∶ (e × ||)
● Encoder: X ∶ (e × 1), U ∶ ( × e), W ∶ ( ×  ), b ∶ ( × 1),
H ∶ ( × 1)
● Decoder: V ∶ (|| ×  ), c ∶ (|| × 1), Ω ∶ (|| × 1)
̂ ∶ (|| × 1)
● Output: Y
●
In the example shown in Figure 10.7, we have || = 4 and  = 2. Selection of embedding layer
width is based on user preference and depends on the characteristics of a given problem. For applications operating in or near real time with conditions changing rapidly, smaller embedding sizes
are acceptable, whereas applications requiring less frequent training may employ larger widths.
When an embedding layer is utilized in an RNN, the elements of the weight matrix E are trained
during the forward and back propagation processes just like the other RNN weight matrices, which
is described next in Section 10.5.1.
10.5 Embedding Layer
Figure 10.7
Vector dimensions in a one-to-one RNN with one embedding layer and one hidden layer.
10.5.1 Forward and Back Propagation with Embedding
Let us now examine the impact of implementing an embedding layer on RNN operations during
the forward and back propagation processes. The first step in forward propagation is the application
of the E matrix to the one-hot-encoded input elements:
xt = E ⋅ xto .
(10.20)
After one-hot-encoded input xto is converted to a dense vector xt by multiplying it by the embedding weight matrix of E, the forward propagation process described by Eqs. (10.3)–(10.7) given
in Section 10.2.2 remain valid for an RNN with an embedding layer. Similarly, back propagation
operations and partial derivatives of L with respect to the RNN weight matrices presented by
Eqs. (10.8)–(10.13) are the same with an embedding layer. However, since we now have an
additional parameter of weight matrix E, we need to compute the partial derivative of L with
respect to E as follows (detailed steps for obtaining this formula are presented in Appendix D):
( t
)
t−1 (

)
∏
𝜕L ∑ ∑
o
†
†
2
2
o
(̂y − yt ) V
W (1 − hj+1 ) (1 − hk ) U xk .
=
(10.21)
𝜕E t=1 k=1 t
j=k
Eqs. (10.14)–(10.15) given in Section 10.2.2 show how to update the RNN parameters with a
learning rate 𝜂 after a back propagation is complete. For an RNN with an embedding layer, we
167
168
10 Recurrent Neural Networks
need to add a similar equation for updating E as follows:
𝜕L
E =E−𝜂 .
𝜕E
(10.22)
10.5.2 A Numerical Example with Embedding
In this section we present a step-by-step description of operations for one epoch of training of an
RNN employing an embedding layer. We will use the same simple example from Section 10.2.1,
whose inputs and outputs are given in Table 10.1. Suppose the network parameters are randomly
initialized as follows:
[
]
[
]
[
]
0.818 −0.360 0.127 −0.036
E=
U = 0.144 −0.232
W = −0.535
0.461 −0.482 0.154 −0.568
[
]†
[ ]
[
]†
c= 0 0 0 0 .
b= 0
V = 0.695 − 0.440 − 0.225 − 0.305
At step t1 , for input x1 = h, we generate prediction ŷ 1 as follows:
[
0.818 −0.360
x1 = E ⋅ x1o =
0.461 −0.482
⎡1⎤
]
] ⎢ ⎥ [
0.818
0.127 −0.036 ⎢0⎥
=
⋅
0.461
0.154 −0.568 ⎢0⎥
⎢0⎥
⎣ ⎦
h1 = tanh(U ⋅ x1 + W ⋅ h0 + b)
(
[
]
)
[
] 0.818
[
] [ ] [ ]
[
]
= tanh 0.144 −0.232 ⋅
+ −0.535 ⋅ 0 + 0 = 0.011
0.461
⎡0⎤ ⎡ 0.008⎤
⎡ 0.695⎤
⎢
⎥ [
] ⎢0⎥ ⎢−0.005⎥
−0.440
⎥
⎥ ⋅ 0.011 + ⎢ ⎥ = ⎢
Ω1 = V ⋅ h1 + c = ⎢
⎢0⎥ ⎢−0.003⎥
⎢−0.225⎥
⎢0⎥ ⎢−0.003⎥
⎢−0.305⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
⎡0.252⎤
⎥
⎢
0.249⎥
ŷ 1 = Θŷ (Ω1 ) = softmax(Ω1 ) = ⎢
.
⎢0.250⎥
⎢0.249⎥
⎦
⎣
The highest value of softmax shows that, for input x1 = h, the prediction is ŷ 1 = h, whereas the
expected output is yo1 = e. The loss L1 is
⎡0⎤ ⎡0.252⎤
⎥
⎢ ⎥ ⎢
0.249⎥
1
= 1.390.
L1 = −yo1 ln(̂y1 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.250⎥
⎢0⎥ ⎢0.249⎥
⎦
⎣ ⎦ ⎣
Continuing with step t = 2, for input x2 = e, we compute ŷ 2 as follows:
[
0.818 −0.360
x2 = E ⋅ x2o =
0.461 −0.482
⎡0⎤
]
] ⎢ ⎥ [
−0.360
0.127 −0.036 ⎢1⎥
=
⋅
−0.482
0.154 −0.568 ⎢0⎥
⎢0⎥
⎣ ⎦
h2 = tanh(U ⋅ x2 + W ⋅ h1 + b)
(
[
]
)
[
] −0.360
[
] [
] [ ]
[
]
= tanh 0.144 −0.232 ⋅
+ −0.535 ⋅ 0.011 + 0 = 0.054
−0.482
10.5 Embedding Layer
⎡ 0.695⎤
⎡0⎤ ⎡ 0.037⎤
⎢
⎥ [
⎢ ⎥ ⎢
⎥
]
−0.440⎥
0
−0.024⎥
Ω2 = V ⋅ h2 + c = ⎢
⋅ 0.054 + ⎢ ⎥ = ⎢
⎢−0.225⎥
⎢0⎥ ⎢−0.012⎥
⎢−0.305⎥
⎢0⎥ ⎢−0.016⎥
⎣
⎦
⎣ ⎦ ⎣
⎦
⎡0.260⎤
⎢
⎥
0.245⎥
.
ŷ 2 = Θŷ (Ω2 ) = softmax(Ω2 ) = ⎢
⎢0.248⎥
⎢0.247⎥
⎣
⎦
Based on the softmax unit, for input x2 = e, the RNN with embedding predicts that ŷ 2 = h,
whereas the expected output is yo2 = l. The loss is
⎡0⎤ ⎡0.260⎤
⎥
⎢ ⎥ ⎢
0.245⎥
0
= 1.395.
L2 = −yo2 ln(̂y2 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.248⎥
⎢0⎥ ⎢0.247⎥
⎦
⎣ ⎦ ⎣
At step t = 3, we continue with input x3 = l and compute ŷ 3 as follows:
[
0.818 −0.360
x3 = E ⋅ x3o =
0.461 −0.482
⎡0⎤
]
] ⎢ ⎥ [
0.127
0.127 −0.036 ⎢0⎥
=
⋅
0.154
0.154 −0.568 ⎢1⎥
⎢0⎥
⎣ ⎦
h3 = tanh(U ⋅ x3 + W ⋅ h2 + b)
(
[
]
)
[
] 0.127
[
] [
] [ ]
[
]
= tanh 0.144 −0.232 ⋅
+ −0.535 ⋅ 0.054 + 0 = −0.046
0.154
⎡0⎤ ⎡−0.032⎤
⎡ 0.695⎤
⎢
⎥ [
] ⎢0⎥ ⎢ 0.020 ⎥
−0.440⎥
⎥
⋅ −0.046 + ⎢ ⎥ = ⎢
Ω3 = V ⋅ h3 + c = ⎢
⎢0⎥ ⎢ 0.010 ⎥
⎢−0.225⎥
⎢0⎥ ⎢ 0.014 ⎥
⎢−0.305⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
⎡0.241⎤
⎢
⎥
0.254⎥
ŷ 3 = Θŷ (Ω3 ) = softmax(Ω3 ) = ⎢
.
⎢0.252⎥
⎢0.253⎥
⎣
⎦
From the softmax unit, for input x3 = l, the RNN predicts that ŷ 3 = e. However, the expected
output is yo3 = l. The loss is
⎡0⎤ ⎡0.2413⎤
⎥
⎢ ⎥ ⎢
0.254 ⎥
0
= 1.379.
L3 = −yo3 ln(̂y3 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢ 0.252 ⎥
⎢0⎥ ⎢ 0.253 ⎥
⎦
⎣ ⎦ ⎣
Finally, at step t = 4, we apply input x4 = l to the RNN with embedding layer and compute ŷ 4 as
follows:
⎡0⎤
[
] ⎢ ⎥ [
]
0.818 −0.360 0.127 −0.036 ⎢0⎥
0.127
x4 = E ⋅ x4o =
⋅
=
0.461 −0.482 0.154 −0.568 ⎢1⎥
0.154
⎢0⎥
⎣ ⎦
169
170
10 Recurrent Neural Networks
h4 = tanh(U ⋅ x4 + W ⋅ h3 + b)
(
[
]
)
[
] 0.127
[
] [
] [ ]
[
]
= tanh 0.144 −0.232 ⋅
+ −0.535 ⋅ −0.046 + 0 = 0.007
0.154
⎡ 0.695⎤
⎡0⎤ ⎡ 0.005⎤
⎢
⎥ [
] ⎢0⎥ ⎢−0.003⎥
−0.440
⎥ ⋅ 0.007 + ⎢ ⎥ = ⎢
⎥
Ω4 = V ⋅ h4 + c = ⎢
⎢−0.225⎥
⎢0⎥ ⎢−0.002⎥
⎢−0.305⎥
⎢0⎥ ⎢−0.002⎥
⎣
⎦
⎣ ⎦ ⎣
⎦
⎡0.251⎤
⎢
⎥
0.249⎥
.
ŷ 4 = Θŷ (Ω4 ) = softmax(Ω4 ) = ⎢
⎢0.250⎥
⎢0.250⎥
⎣
⎦
Output from softmax indicates that, for entering the second l as an input, the RNN predicts
that ŷ 4 = h, whereas the expected output is yo4 = o. The loss is
⎡0⎤ ⎡0.251⎤
⎢ ⎥ ⎢
⎥
0
0.249⎥
L4 = −yo4 ln(̂y4 ) = − ⎢ ⎥ ln ⎢
= 1.388.
⎢0⎥ ⎢0.250⎥
⎢1⎥ ⎢0.250⎥
⎣ ⎦ ⎣
⎦
The total loss after four steps is
L=

∑
Lt = 1.390 + 1.395 + 1.379 + 1.388 = 5.552.
t=1
Let us now continue with the back-propagation process starting with computing the gradient of
L with respect to predicted output ŷ from Eq.(10.8) as follows:
⎡0.252⎤ ⎡0⎤ ⎡ 0.252⎤
⎢
⎥ ⎢ ⎥ ⎢
⎥
𝜕L1 𝜕 ŷ 1
0.249⎥ ⎢1⎥ ⎢−0.751⎥
= (̂y1 − yo1 ) = ⎢
−
=
⎢0.250⎥ ⎢0⎥ ⎢ 0.250⎥
𝜕 ŷ 1 𝜕Ω1
⎢0.249⎥ ⎢0⎥ ⎢ 0.249⎥
⎣
⎦ ⎣ ⎦ ⎣
⎦
⎡0.260⎤ ⎡0⎤ ⎡ 0.260 ⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L2 𝜕 ŷ 2
0.245⎥ ⎢0⎥ ⎢ 0.245 ⎥
=
−
= (̂y2 − yo2 ) = ⎢
⎢0.248⎥ ⎢1⎥ ⎢−0.7522⎥
𝜕 ŷ 2 𝜕Ω2
⎢0.247⎥ ⎢0⎥ ⎢ 0.247 ⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
⎡0.241⎤ ⎡0⎤ ⎡ 0.241⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L3 𝜕 ŷ 3
0.254⎥ ⎢0⎥ ⎢ 0.254⎥
=
−
= (̂y3 − yo3 ) = ⎢
⎢0.252⎥ ⎢1⎥ ⎢−0.748⎥
𝜕 ŷ 3 𝜕Ω3
⎢0.253⎥ ⎢0⎥ ⎢ 0.253⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
⎡0.251⎤ ⎡0⎤ ⎡ 0.251⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L4 𝜕 ŷ 4
0.249⎥ ⎢0⎥ ⎢ 0.249⎥
.
=
−
= (̂y4 − yo4 ) = ⎢
⎢0.250⎥ ⎢0⎥ ⎢ 0.250⎥
𝜕 ŷ 4 𝜕Ω4
⎢0.250⎥ ⎢1⎥ ⎢−0.750⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
10.5 Embedding Layer
We now compute the gradients of L with respect to V, c, W, U and E using Eqs. (10.9), (10.10),
(10.11), (10.12) and (10.21), respectively, as follows:
⎡ 0.008⎤


⎢
⎥
−0.005⎥
𝜕L ∑ 𝜕Lt ∑
†
(̂yt − yot ) ⋅ ht = ⎢
=
=
⎢−0.001⎥
𝜕V
𝜕V
t=1
t=1
⎢−0.001⎥
⎣
⎦
⎡ 1.005⎤

⎢
⎥
∑
−0.003⎥
𝜕L
(̂yt − yot ) = ⎢
=
=
⎢−1.001⎥
𝜕c
𝜕c
t=1
t=1
⎢−0.002⎥
⎣
⎦

∑
𝜕Lt
( t
)
t−1 (

)
∑
∑
∏
[
]
𝜕L
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) hk−1 = −0.0082
𝜕W
t=1
k=1
j=k
( t
)
t−1 (

)
∏
[
]
𝜕L ∑ ∑
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) xk = 0.214 0.098
𝜕U
t=1
k=1
j=k
)
( t
t−1 (

)
∏
[
]
𝜕L ∑ ∑
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) = 0.703
𝜕b
t=1
k=1
j=k
( t
)
t−1 (

)
∏
𝜕L ∑ ∑
o
†
†
2
2
o
=
(̂y − yt )V
W (1 − hj+1 ) (1 − hk )Uxk
𝜕E t=1 k=1 t
j=k
[
]
0.054 0.011 0.036 0
=
.
−0.087 −0.018 −0.058 0
From Eqs. (10.14)–(10.18) and Eq. (10.22), we now update RNN parameter values for a learning
rate of 𝜂 = 0.1 (arbitrarily chosen), as follows:
⎡ 0.0075⎤ ⎡ 0.694⎤
⎡ 0.695⎤
⎥
⎥ ⎢
⎢
⎥
⎢
−0.0051⎥ ⎢−0.440⎥
𝜕L ⎢−0.440⎥
=
− 0.1 ⎢
=
Vnew = V − 𝜂
⎢−0.0013⎥ ⎢−0.224⎥
𝜕V ⎢−0.225⎥
⎢−0.0011⎥ ⎢−0.305⎥
⎢−0.305⎥
⎦
⎦ ⎣
⎣
⎦
⎣
⎡ 1.0052⎤ ⎡−0.1005⎤
⎡0⎤
⎥
⎥ ⎢
⎢
⎢ ⎥
−0.0025⎥ ⎢ 0.0002⎥
𝜕L ⎢0⎥
cnew = c − 𝜂
=
− 0.1 ⎢
=
⎢−1.0011⎥ ⎢ 0.1001⎥
𝜕c ⎢0⎥
⎢−0.0016⎥ ⎢ 0.0002⎥
⎢0⎥
⎦
⎦ ⎣
⎣
⎣ ⎦
Wnew = W − 𝜂
Unew = U − 𝜂
bnew = b − 𝜂
[
]
[
] [
]
𝜕L
= −0.535 − 0.1 −0.008 = −0.534
𝜕W
]
[
] [
]
𝜕L [
= 0.144 −0.232 − 0.1 0.214 0.098 = 0.123 −0.241
𝜕U
[
] [
]
𝜕L [ ]
= 0 − 0.1 0.703 = −0.070
𝜕b
171
172
10 Recurrent Neural Networks
𝜕L
Enew = E − 𝜂
𝜕E
[
]
[
]
0.818 −0.360 0.127 −0.036
0.054 0.011 0.036 0
=
− 0.1
0.461 −0.482 0.154 −0.568
−0.087 −0.018 −0.058 0
[
]
0.812 −0.361 0.123 −0.036
=
,
0.470 −0.481 0.160 −0.568
where subscript new is used to distinguish between the newly updated and previous values of the
RNN parameters. At this point, training continues with another forward propagation followed by a
back propagation with the updated values. In epoch 2, the total loss L was found to be 5.311, reduced
from its previous value of 5.552 (not shown here for space constraints). When we continued training
this RNN, the total error was further reduced to 2.927 at 50 epochs and to 0.880 at 1000 epochs.
10.6 Concluding Remarks
In this chapter, we introduced RNNs, which are powerful tools that capture complex relations
among elements of input sample data for realistic applications such as chatbots or machine
translation. The main strength of RNNs stems from the fact that, through their hidden states,
they retain information generated at earlier steps from previously applied inputs. This capability
enables a form of memory generated by earlier inputs and makes RNNs excellent candidates to
handle sequence-based information. Another important feature of RNNs is that they can process
variable-length input sequences thanks to the recursive nature of hidden states, which is presented
in Chapter 11 implementing a simple chatbot using a many-to-many RNN.
With the help of a simplistic example, we provided step-by-step descriptions for RNN training
operations, namely obtaining gradients of a loss function with respect to partial derivatives of its
parameters and mechanisms for updating weight matrices during forward- and back-propagation
training steps. We examined the impact of embedding layers as a means of reducing input data
dimensions to reduce the computational cost of the training process. However, due to their ability
to deal with sequences of inputs, an output at each step needs the information from earlier
steps, which disqualifies RNNs from operating in parallelized fashion and, hence, makes them
computationally inefficient.
Although very useful for modelling sequences, other significant disadvantages of RNNs are
so-called vanishing and exploding gradient problems, which lead to long-term
memory problems in RNNs [95]. A vanishing gradient problem is observed when the element
values in weight matrices become so small that learning either becomes very slow or stops
altogether. Conversely, if weight values become large, they can create divergent solutions over a
search space, which is referred to as an exploding gradient problem. Different modifications for
RNNs have been proposed to handle such situations, including LSTMs [8], transformers [9] and
attention mechanisms [96]. The reader is encouraged to study these solutions after digesting the
materials provided in Chapters 10, 11, 12 and 13 of this book.
10.7 Exercises for Chapter 10
1) What is the impact of increasing the learning rate 𝜂 on the prediction power of an RNN? Derive
an approximate formula for it.
2) What is the impact of decreasing the learning rate 𝜂 on the prediction power of an RNN? Derive
an approximate formula for it.
10.7 Exercises for Chapter 10
3) What is the impact of increasing the number of epochs on the prediction power of an RNN?
Derive an approximate formula for it.
4) What is the impact of decreasing the number of epochs on the prediction power of an RNN?
Derive an approximate formula for it.
5) Derive the weight matrix dimensions for a three-cell RNN using a single hidden layer (use the
two-cell RNN given in Figure 10.6 as a guide).
6) The introduction of embedding layers improves the computational cost of training in RNNs.
Demonstrate this fact by deriving approximate order of magnitude formulas for the computational cost of one epoch for an RNN with and without embedded layers. Keep all other parameters fixed to demonstrate the impact of embedding layers.
7) Derive the matrix dimensions of weight matrices for a three-cell RNN using two hidden layers
(use the RNN given in Figure 10.7 as a guide).
173
175
11
RNNs for Chatbot Implementation
11.1 Introduction
The imitation game, proposed by Alan Turing in 1950 [2], states that if a machine (e.g. a
computer) can answer any question put to it using the words that an ordinary human would, then
we may call this machine intelligent. The Turing test [97] is based on this game, whose
goal is to determine if the answers provided by the machine are indistinguishable from those by
a human. The Turing test, where a machine tries to impersonate a human, is considered one of
the earliest attempts at defining artificial intelligence, although it is debatable that answers to such
questions requires any real intelligence other than simple search techniques based on previous
conversations.
A modern implementation of this experiment would employ a chatbot [98], which is a program
that mimics a conversation with a human using natural language interactions (e.g. text or voice)
[99, 100]. Current chatbots are typically online and capable of maintaining a conversation in a
natural language based on input stimulus provided by the user. Recently chatbots have evolved to
bring a significant societal impact by potentially replacing humans in many professional settings,
including answering services and customer service operations.
In this chapter, we present an implementation of a simplistic chatbot based on an RNN, which
was introduced in Chapter 10. This is a fairly difficult problem for other AI tools such as ANNs
and CNNs since they do not look for a temporal association between different elements of an
input sequence but rather focus on current input/output pairs without taking into account previously applied inputs and their respective outputs (i.e. they lack a hidden state). Another difficulty in chatbot implementation is the asymmetric relationship between the lengths of input
and output sequences. For example, a user input of how is the weather this afternoon?
send to a chatbot impersonating a personal assistant may generate a response of it will be
cloudy and chilly, whereas a shorter input of will it rain today? may generate an output with a different length, such as yes it is likely. RNNs that can handle variable-length
input sequences and generate variable-length output sequences are called many-to-many RNNs
[87], as studied below in Section 11.2. Python and Matlab scripts implementing a simplistic chatbot
using a many-to-many RNN are given in Sections 11.4 and 11.5, respectively.
11.2 Many-to-Many RNN Architecture
Applications such as chatbots require RNNs that operate in a special way as follows. Until a user
completes entering a sequence of inputs X1 , · · · , Xk−1 , the RNN does not generate any external
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
176
11 RNNs for Chatbot Implementation
Figure 11.1 A many-to-many RNN with multiple hidden layers used in a chatbot implementation, where
no outputs are generated until the user completes entering an input sequence of x1 , · · · , xk until step k,
after which external outputs are generated and fed back into the RNN as inputs (i.e. outputs ŷ k , ŷ k+1 , · · · are
applied to the RNN as inputs xk+1 , xk+2 , · · ·, respectively).
outputs but only computes its hidden states h1 , · · · , hk−1 . At step k, the user enters Xk as the last
input in the sequence, for which the RNN generates an output Ŷ k . At step (k + 1), this output Ŷ k
is fed back into the RNN as an input Xk+1 to generate output Ŷ k+1 . Similarly, at step (k + 2), Ŷ k+1
is applied to the RNN as an input Xk+2 , and so on. Typically, the input Xk at which the RNN starts
generating outputs is a special input character such as an end-of-sequence indicator. Figure 11.1
illustrates an example for a many-to-many RNN with multiple hidden layers, where no outputs
are generated until the user completes entering an input sequence at step k, after which external
outputs are generated and fed back into the RNN as inputs (i.e. outputs Ŷ k , Ŷ k+1 , · · · are applied to
the RNN as inputs of Xk+1 , Xk+2 , · · ·, respectively).
11.3 A Simple Chatbot
In this chapter, we implement a simplistic chatbot using a many-to-many RNN employing hidden
and embedding layers. This chatbot will be trained with 500 sample query and response pairs taken
from the internet, which contain 565 unique words. For a real-life chatbot implementation, tens of
millions of pairs will be needed. Figure 11.2 shows a subset of sample data containing queries and
responses used in training the RNN. In the training data, a user query is given as
<start> hi, how are you doing ? <end>
for which the sample data defines a response as
<start> I am fine. how about yourself ? <end>
where special words (or tokens) of <start> and <end> are added to each sample to indicate the
beginning and end, respectively, for each query and response pair. With the help of these special
tokens, the RNN will know that when it reads <start> in a user query, it starts generating hidden
states without external outputs until it reads <end>, after which the RNN starts generating external
outputs and using each output as an input for the next step, as described previously. Once the RNN
11.3 A Simple Chatbot
(a)
(b)
Figure 11.2 A subset of samples used for training in Python and Matlab implementations of the
many-to-many RNN presented in Sections 11.4 and 11.5: (a) user queries, and (b) corresponding answers.
177
178
11 RNNs for Chatbot Implementation
Figure 11.3 Weight matrix dimensions in a many-to-many RNN with 256 hidden layers used in chatbot
implementation for a dictionary size of 565 words.
generates a won output containing <end>, it stops generating outputs. Once the RNN is trained,
the user may enter arbitrary queries as long as they contain only the words present in the alphabet.
For example, the following user query is not part of the training data:
<start> how was your day today ? <end>
For this user query, the RNN implemented by the Python script in Section 11.4 generates the
following response, which is also not present in any of the queries used in the training data:
<start> i am a movie ?. <end>
Of course the response does not make any sense semantically since we are using very few data
samples and training the RNN for only a few epochs; however, it is remarkable that this simplistic
RNN example is capable of handling unique queries and generating unique responses. It is clear
that if we use millions of data samples and train the RNN for tens of thousands of epochs, the
responses will significantly improve.
As described in Section 10.4 in Chapter 10, it is customary to include multiple hidden layers for
realistic RNN implementations to improve the effectiveness of training, where hidden layers offer
parallel and simultaneous searches at the expense of increasing weight matrix dimensions.
In Figure 11.3, we show the matrix dimensions for a many-to-many RNN with 256 hidden layers,
for an alphabet size of 565 unique words as used in our Python and Matlab implementations in
Sections 11.4 and 11.5 below.
11.4 Python Script for a Chatbot Using an RNN
Figure 11.4 Weight matrix dimensions in a many-to-many RNN with 256 hidden layers and an embedding
layer of size 128 used in chatbot implementation for a dictionary size of 565 words.
As presented in Section 10.5 in Chapter 10, adopting an embedding layer in RNNs generates
compact representations of input vectors and, hence, significantly reduces the computational
cost of the training process. Figure 11.4 shows the weight matrix dimensions for an RNN with
256 hidden layers and an embedding matrix E of size (128 × 565) with a 565-word dictionary used
in the Python and Matlab scripts. Recall from Eq. (10.20) in Chapter 10 that for the embedding
layer the matrix multiplication is defined as X = E ⋅ X o , where X o is the one-hot-encoded input
vector.
11.4 Python Script for a Chatbot Using an RNN
In this section we present a sample Python script implementing the simplistic chatbot described in
Section 11.3 as a many-to-many RNN. This RNN is constructed with the same architecture shown
in Figure 11.4 with 256 hidden layers and an embedding size of 128 for processing user queries from
a dictionary of 565 unique words. After digesting the sample code presented below, the reader is
encouraged to experiment by changing the parameters for the RNN architecture and training in
order to gain a good understanding of the impact of each one on the RNN prediction accuracy.
In line 23, the number of samples to train the RNN is set as 500. In line 24, the batch size is set
to 16, which is a training parameter for the number of input samples to be handled at one time per
179
180
11 RNNs for Chatbot Implementation
epoch. For example, if there are 1000 data samples and the batch size is 100, during one epoch, the
RNN will process 10 batches, each of which has 100 input samples. By processing sample data in
small chunks, we limit the matrix dimensions during training (in this case to 16), whereas handling
all input samples at once would require manipulating matrices with much larger dimensions (in
this case 1000) with a much higher computational cost. In line 25, the number of hidden layers is
set to 256, which implies that the RNN utilizes 256 encoders operating in parallel. In this case, if
embedding is not used, the dimensions of U would be (256 × 565), where we use  = 500 samples
containing a total of || = 565 unique words. In line 26, we set the number of epochs as 1000,
which is the total number of forward and back propagations used training the RNN. In line 27,
the embedding layer size is set as 128, which means that dimensions of E will be (128 × 565). With
embedding, a single input sample size is reduced from (565 × 1) to (128 × 1) as shown in Figure 11.4.
Embedding reduces the dimensions of U from (256 × 565) to (256 × 128).
In lines 29–41, the function named tokenize accepts the data sample set as its input, converts
the words into numeric tokens and adds padding based on a user-defined maximum sequence
length. Padding [101] is a useful technique in handling variable-length input and output
sequences, where all inputs and outputs are padded with blank characters to make them appear
to have equal lengths. In line 33, an alphabet called lang_tokenizer is created based on input
samples, which is encoded in line 37 using sparse one-hot encoding [92]. In lines 39 and
40, padding is added to the encoded alphabet. In lines 43–51, the load_dataset function, whose
inputs are the sample inputs and the expected outputs, generates the encoded and padded sample
inputs called input_tensor (i.e. the X set), the expected outputs called target_tensor (i.e.
the Y set) and the alphabets for both inputs and expected outputs called lang_tokenizer.
In lines 53–64, the create_rnn_model function creates the RNN model. In line 58, an embedding layer using the alphabet size and embedding dimension is created; similary, an encoder using
the number of hidden layers is created in line 60 and a decoder using the size of expected output
alphabet in line 62.
In lines 66–92, the predict function is implemented, which generates a response for a given
user input sentence as follows. In line 69, special start and end words are added. In line 71, the words
are converted to numeric vectors and, in lines 72–74, padding is added. In line 76, the input queries
are converted to the tensor flow data type to be used later in the RNN model on line 78. A vector
called prediction containing the outputs of the softmax unit is also created at line 78. In line
83, the index of the highest probability value in the prediction is found by the tf.argmax function.
In lines 88 and 89, the tokens are converted to words. After removing the tags, the function returns
the predicted output sentence.
In lines 94–104, the input samples are read from a file and split into sample inputs and
corresponding outputs, which are then loaded into matrices called input_tensor_train and
target_tensor_train, respectively, in line 103. An RNN model is created in line 110 using
the parameters defined above, namely the alphabet size, embedding dimension and number of
hidden layers. In line 113, a tensor flow object called optimizer, which uses gradient descent
for optimization (i.e. the Adam option), is created. In lines 114–115, the loss function is defined
as a sparse cross-entropy loss [93]. In lines 127 and 128, the RNN is trained using the
input and output matrices, the number of epochs, the batch size and the choice for the shuffle
operation (if true, the order of input samples is shuffled at each epoch) and stored into an object
called model. The remaining lines are for a simplistic user interface to either train a new RNN or
use a previously trained one, to ask queries either directly on screen or from a previously recorded
file and for storing the current results.
11.4 Python Script for a Chatbot Using an RNN
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
# The City College of New York, City University of New York
# Written by: Hasan Suca Kayman
# June 2024
# chatbot_RNN is a simple chatbot that uses a Recurrent Neural Network
# to generate responses to user queries. Training matrials is a set of
# dialogs taken from the text messages arbitrarily chosen from the Internet.
# Importing necessary libraries
import tensorflow as tf
from datetime import datetime
import os
import time
# Setting a random seed for reproducibility
seed = 301
tf.keras.utils.set_random_seed(seed)
# Example queries file
queries_path = 'questions.txt'
# Data input file
data_path = 'cleaned_dialogs.txt'
noof_samples = 500
batch_size = 16
noof_hidden_layers = 256
noof_epochs = 1000
embedding_dim = 128
# Tokenize a list of sentences and pad them to ensure equal length
# Convert words into numeric tokens and add padding based on max length
def tokenize(lang):
# Create a tokenizer
lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
# Fit tokenizer to the given text
lang_tokenizer.fit_on_texts(lang)
# Convert text to sequences of numbers
tensor = lang_tokenizer.texts_to_sequences(lang)
# Pad sequences
tensor = \
tf.keras.preprocessing.sequence.pad_sequences(tensor,padding='post')
return tensor, lang_tokenizer
# Load a dataset and tokenize both input and target (expected) languages
def load_dataset(inp_lang, targ_lang):
# Unpack the given data into target and input languages
lang = inp_lang + targ_lang
# Tokenize both the input and target languages
data, lang_tokenizer = tokenize(lang)
input_tensor, target_tensor = data[:len(inp_lang)], data[len(inp_lang):]
# Return tokenized inputs, targets, and their tokenizers
return input_tensor, target_tensor, lang_tokenizer
# Create a simple RNN model with specified parameters
def create_rnn_model(alph_size, embedding_dim, rnn_units):
# Define a sequential model with embedding, RNN, and output layer
model = tf.keras.Sequential([
# Embedding layer
tf.keras.layers.Embedding(alph_size, embedding_dim),
# Simple RNN layer
tf.keras.layers.SimpleRNN(rnn_units, return_sequences=True),
# Dense output layer
tf.keras.layers.Dense(alph_size)
])
return model
# Given an input sentence, generate a response using the model
def predict(sentence, model, tokenizer, max_length=20):
# Preprocess the input sentence
preprocessed_sentence = '<start> ' + sentence.lower() + ' <end>'
# Tokenize the sentence and pad it to the max_length
input_sequence = tokenizer.texts_to_sequences([preprocessed_sentence])
input_tensor = \
181
182
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
11 RNNs for Chatbot Implementation
tf.keras.preprocessing.sequence.pad_sequences(input_sequence, \
maxlen=max_length, padding='post')
# Convert to a tensor and reshape for the RNN model
input_tensor = tf.convert_to_tensor(input_tensor)
# Generate predictions
prediction = model.predict(input_tensor, verbose=0)
# Extract the most likely tokens and convert them back to words
predicted_sequence = ['<start> ']
for pred in prediction[0]:
# Get the token with the highest probability
predicted_id = tf.argmax(pred, axis=-1).numpy()
# Stop if we reach the end token
if tokenizer.index_word.get(predicted_id, '') == " <end>":
break
# Convert token to word
predicted_word = tokenizer.index_word.get(predicted_id, '')
predicted_sequence.append(predicted_word)
# Return the original input and the predicted output and removing tags
result=' '.join(predicted_sequence).split("<start>")[-1].split("<end>")[0]
return result, sentence
# Read dialogs from a text file
file = open(data_path,'r').read()
# Split the dialog into questions and answers
qna_list = [f.split('\t') for f in file.split('\n')][:-1]
questions = [x[0] for x in qna_list][:noof_samples+1]
answers = [x[1] for x in qna_list][:noof_samples+1]
# Load and tokenize the dataset
input_tensor_train, target_tensor_train, lang_tokenizer = \
load_dataset(questions, answers)
# Set the unit count for alphabet sizes
alph_size = len(lang_tokenizer.word_index)+1
# Create the RNN model with specified parameters
model = create_rnn_model(alph_size, embedding_dim, noof_hidden_layers)
# Define the optimizer and loss function for training
optimizer_a = tf.keras.optimizers.Adam()
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True, reduction='none')
model.compile(optimizer=optimizer_a, loss=loss_object)
# prompt user to train or load an RNN model
print('\nOPTIONS:')
print('1 - Train a new RNN model\n2 - Load an existing model')
option = input('\nSelect an option by entering a number: \n')
if option == '1':
# OPTION 1: TRAIN A NEW RNN MODEL
print('\n********* NOW TRAINING A NEW RNN MODEL *********')
model.fit(input_tensor_train, target_tensor_train, epochs=noof_epochs, \
batch_size = batch_size, shuffle=True)
print('\n\n********** RNN training complete **********\n\n')
elif option == '2':
# OPTION 2: LOAD RNN MODEL FROM FILE
message = 'Enter the file name of the RNN Model you want to load: \n'
load_file = input(message)
# load the RNN model from load_file
model.load_weights(load_file)
print('\n\n****** SUCCESSFULLY LOADED RNN MODEL ', load_file,'******')
# prompt user to test or save an RNN model
option_list = ['1','2','3','4']
while option != '4':
option = ' '
print('\nOPTIONS:')
11.5 Matlab Script for a Chatbot Using an RNN
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
print('1 - Test this RNN model with input')
print('2 - Test queries given in a file')
print('3 - Save existing model and logs')
print('4 - Exit')
option = input('\nSelect an option by entering a number: \n')
if option == '1':
# OPTION 1: TEST MODEL
msg = 'Enter your input: '
user_input = input(msg)
result, sentence = \
predict(user_input, model, lang_tokenizer, alph_size)
print('Question: {}'.format(sentence))
print('Predicted Output: {}'.format(result))
elif option == '2':
# OPTION 2: TEST QUERIES
# List of queries for the model to respond to
file = open(queries_path,'r').read()
queries = [question for question in file.split('\n')]
for index,query in enumerate(queries):
result, sentence = predict(query, model, lang_tokenizer, \
alph_size)
print('{}.Question: {}'.format(index+1,sentence))
print('{}.Predicted Output: {}'.format(index+1,result))
elif option == '3':
# OPTION 3: SAVE LOGS AND MODEL
times = datetime.now().strftime("%Y_%m_%d_%H_%M_%S")
model_path = 'RNN_MODEL_N{}_B{}_U{}_EP{}_EM{}_{}.h5'.format(
noof_samples, batch_size, noof_hidden_layers,
noof_epochs, embedding_dim,times
)
model.save_weights(model_path)
print("Succesfully Saved {}".format(model_path))
A small set of outputs generated by the chatbot implemented by the above Python script is given
in Figure 11.5. The user queries taken from the sample internet data and the responses generated
by our chatbot are shown in Figure 11.5a. The user queries that are not in the sample data but use
the words in the dictionary and the corresponding responses generated by the Python chatbot are
in Figure 11.5b. We see that most of the responses for both sets of queries appear to be adequate
considering the limitations in sample size, the number of epochs, and the number of hidden and
embedding layers.
11.5 Matlab Script for a Chatbot Using an RNN
In this section we present a sample Matlab implementation of the simplistic chatbot for the same
many-to-many RNN described in Section 11.3. The Matlab script given below uses similar structure,
variables and functions as in the Python script given in Section 11.4 so that the reader can easily
compare them.
In lines 16 to 20, RNN parameters such as the number of samples, batch size, hidden layers,
epochs and dimension of embedding layer are set. In lines 22 to 33, sample data is read and
converted to a special format called cell. In lines 35 to 44, sample data is split into two groups
as queries and responses. In line 48, the load_dataset function is called to encode the
queries and responses using so-called sparse one-hot encoding [92], which returns four
variables, namely input_tensor_train, target_tensor_train, lang_tokenizer,
and rev_tokenizer representing the inputs (i.e. the set of X), the expected outputs (i.e. the set
of Y ), and their encoding and decoding mappings, respectively. In lines 54 and 58, alphabet size
is set and the type for output is adjusted, respectively. In line 67, the RNN architecture is defined
183
184
11 RNNs for Chatbot Implementation
(a)
(b)
Figure 11.5 Sample outputs generated by the Python script presented in Section 11.4: (a) user queries and
responses taken from the sample training data and the responses generated by the chatbot, and (b) unique
user queries and the responses generated by the chatbot.
by calling the function named create_rnn_model with parameters of input layer, embedding
layer and number of hidden layers. In line 70, the training options are set, including gradient
descent as the optimization method (i.e. selecting Adam) and the shuffle option (i.e. changing the
order of samples after each epoch to avoid possible bias in them).
A rudimentary user interface is implemented between lines 73 and 136. When a user selects
option 1 in line 80, a new RNN is trained in lines 83 and 84. For option 2, a previously trained RNN
is loaded in line 89. Once a new or previously trained RNN is loaded, if the user selects option 1,
the function named predictResponse is called in line 107 to generate a prediction for a single
user query entered on screen. Similarly, for selecting option 2, in line 119, predictResponse
is called for each of the queries that are stored in the questions.txt file. Option 3 saves the
current RNN, user queries and responses in lines 123–131. Entering 4 as the option will terminate
the program in line 134.
The function named load-dataset is implemented between lines 138 and 157. It merges
queries and outputs to obtain the object called lang in line 141 and calls the tokenDictionary
11.5 Matlab Script for a Chatbot Using an RNN
function to map lang into inputs and outputs. In lines 151 and 152, queries and outputs are
mapped into input_tensor and target_tensor, respectively. In lines 159–170, the function
named TokenDictionary is implemented. It counts the unique words and creates a dictionary.
In line 168, it generates an encoded table for this dictionary and, in line 169, a decoded table.
In lines 172–181, the function named create_rnn_model uses the input, embedding hidden
layers, many-to-many RNN and decoder defined above to form an RNN object called layers. In
lines 183–196, the predictResponse function is given, which adds the start and end tokens to
a given sentence, encodes and pads it, and then uses classify to generate a prediction. Using
decoder, it converts the prediction into a word in dictionary and, after eliminating the start and
end tokens, returns it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
% The City College of New York, City University of New York
% Written by: Hasan Suca Kayman
% July 2024
% chatbot_RNN is a simple chatbot that uses a Recurrent Neural Network
% to generate responses to user queries. Training materials is a set of
% dialogs taken from text messages arbitrarily chosen from the Internet.
% Setting a random seed for reproducibility
rng(301)
% Example queries file
queries_path = 'questions.txt';
% Data input file
data_path = 'cleaned_dialogs.txt';
noof_samples = 500;
batch_size = 16;
noof_hidden_layers = 256;
noof_epochs = 400;
embedding_dim = 128;
% Read dialogs from a text file
fileID = fopen(data_path, 'r');
textData = textscan(fileID, '%s', 'Delimiter', '\n');
fclose(fileID);
% Each line from input file is a cell array of strings
textData = textData{1};
% Converting data type to cell
qna_list = cell(size(textData));
questions = cell(noof_samples+1, 1);
answers = cell(noof_samples+1, 1);
% Split each line into questions and answers
for i = 1:numel(textData)
qna_list{i} = strsplit(textData{i}, '\t');
end
% Extract questions and answers
for i = 1:noof_samples+1
questions{i} = qna_list{i}{1};
answers{i} = qna_list{i}{2};
end
% Load and tokenize the dataset
[input_tensor_train, target_tensor_train, lang_tokenizer, ...
rev_tokenizer] = load_dataset(questions, answers);
% Set the unit count for alphabet sizes
alph_size = 0;
for i = 1:noof_samples+1
if max(target_tensor_train{i}(:)) > alph_size
alph_size = max(target_tensor_train{i}(:));
end
185
186
11 RNNs for Chatbot Implementation
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
end
% Add one for padding token
alph_size = alph_size + 1;
% Converting data type to categorical
for idx = 1:numel(target_tensor_train)
target_tensor_train{idx} = categorical(target_tensor_train{idx},...
0:alph_size-1);
end
% Create the RNN model with specified parameters
layers = create_rnn_model(alph_size, embedding_dim, noof_hidden_layers);
% Defining training options
options = trainingOptions('adam', 'MaxEpochs', noof_epochs, ...
'MiniBatchSize', batch_size,'Shuffle', 'every-epoch');
% User interaction for training or loading a model
disp('OPTIONS:');
disp('1 - Train a new RNN model');
disp('2 - Load an existing model');
option = input('Select an option by entering a number: ', 's');
switch option
case '1'
% OPTION 1: TRAIN A NEW RNN MODEL
disp('********* NOW TRAINING A NEW RNN MODEL *********');
model = trainNetwork(input_tensor_train, target_tensor_train,...
layers, options);
disp('********** RNN training complete **********');
case '2'
% OPTION 2: LOAD RNN MODEL FROM FILE
modelFileName = input('Enter the RNN Model to load: ', 's');
load(modelFileName, 'model');
disp(['****** SUCCESSFULLY LOADED RNN MODEL ', ...
modelFileName,'******']);
end
% User interaction for testing or saving a model
while true
disp('OPTIONS:');
disp('1 - Test this RNN model with input');
disp('2 - Test queries given in a file');
disp('3 - Save existing model and logs');
disp('4 - Exit');
option = input('Select an option by entering a number: ', 's');
switch option
case '1'
% OPTION 1: TEST MODEL
user_input = input('Enter your input: ', 's');
predicted_sentence = predictResponse(model, ...
lang_tokenizer, rev_tokenizer, user_input);
disp(['Predicted Output: ', predicted_sentence]);
case '2'
% OPTION 2: TEST QUERIES
fileID = fopen(queries_path, 'r');
queries = textscan(fileID, '%s', 'Delimiter', '\n');
fclose(fileID);
queries = queries{:};
for i = 1:length(queries)
query = queries{i};
disp(['Query ', num2str(i), ': ', query]);
predicted_sentence = predictResponse(model, ...
lang_tokenizer, rev_tokenizer, query);
disp(['Predicted Output: ', predicted_sentence]);
end
case '3'
% OPTION 3: SAVE LOGS AND MODEL
dtw = datestr(now, 'yyyy_mm_dd_HH_MM_SS');
model_path=sprintf('RNN_model_N%d_B%d_U%d_EP%d_EM%d_%s.mat',...
noof_samples, batch_size, noof_hidden_layers, ...
noof_epochs, embedding_dim, dtw);
11.5 Matlab Script for a Chatbot Using an RNN
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
save(model_path, 'model');
disp(['Successfully saved to',model_path]);
case '4'
disp('Exiting program');
break;
end
end
function [input_tensor, target_tensor, lang_tokenizer, ...
reverseTokenizer] = load_dataset(inp_lang, targ_lang)
% Concatenate inputs and targets for combined tokenization
lang = [inp_lang; targ_lang];
% Tokenize both the input and target languages
[lang_tokenizer,reverseTokenizer] = tokenDictionary(lang);
% Initialize cell arrays for storing sentences
input_tensor = cell(length(inp_lang), 1);
target_tensor = cell(length(targ_lang), 1);
% Encode all of input and target data
for i = 1:length(inp_lang)
input_tensor{i} = encoder(lang_tokenizer, inp_lang{i}, 25);
target_tensor{i} = encoder(lang_tokenizer, targ_lang{i}, 25);
end
% Transpose them to correct dimensions
input_tensor = input_tensor';
target_tensor = target_tensor';
end
function [tokenizer,reverseTokenizer] = tokenDictionary(texts)
% Create a tokenizer
joined_texts = join(string(texts));
% Convert text to sequences of numbers
unique_words = unique(strsplit(joined_texts));
% Pad sequences
unique_words = ['<pad>', unique_words];
indices = 1:numel(unique_words);
% Fit tokenizer to the given text
tokenizer = containers.Map(unique_words, indices);
reverseTokenizer = containers.Map(num2cell(indices), unique_words);
end
% Create a simple RNN model with specified parameters
function layers = create_rnn_model(alph_size, embedding_dim, rnn_units)
layers = [ ...
sequenceInputLayer(1)
wordEmbeddingLayer(embedding_dim, alph_size) % Embedding layer
lstmLayer(rnn_units, 'OutputMode', 'sequence') % an RNN layer
fullyConnectedLayer(alph_size)
softmaxLayer % softmax layer
classificationLayer];
end
function predicted_sentence = predictResponse(model, lang_tokenizer, ...
rev_tokenizer, user_input)
% Preprocess the input sentence
preprocessed_sentence = ['<start> ', lower(user_input), ' <end>'];
% Tokenize the sentence and pad it to the max_length
input_sequence = encoder(lang_tokenizer, preprocessed_sentence, 25);
% Generate predictions
predictions = classify(model, input_sequence);
% Convert tokens back to words
predicted_words = decoder(rev_tokenizer, predictions);
% Remove start and end tags and join into a single sentence
predicted_sentence = strrep(predicted_words,'<start> ','');
predicted_sentence = strrep(predicted_sentence,' <end>','');
end
function text = decoder(reverseTokenizer, sequence)
% Sequences that include words are converted to sparse one-hot encoding
% word by word:
words = cell(1, numel(sequence));
187
188
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
11 RNNs for Chatbot Implementation
for i = 1:numel(sequence)
intVal = double(string(sequence(i)));
if intVal ∼= 0 && reverseTokenizer.isKey(intVal)
words{i} = reverseTokenizer(intVal);
else
words{i} = '';
end
end
text = strjoin(words);
end
function sequence = encoder(tokenizer, text, max_length)
% Word by word, sparse one-hot encoding is applied to sequences that
% include words:
words = strsplit(text);
sequence = zeros(1, max_length);
for i = 1:length(words)
if isKey(tokenizer, words{i})
sequence(i) = tokenizer(words{i});
end
end
end
Figure 11.6 shows the outputs generated by the chatbot implemented by the above Matlab script.
For the user queries taken from the sample internet data, the responses generated by the chatbot
are shown in Figure 11.6a. We observe that the responses are quite similar to the outputs given
in the sample training data. For queries that are not in the sample data (but use the words in
the samples), the responses generated by the chatbot are in Figure 11.6b. We can see that these
queries are more difficult for the chatbot to process. Some of the responses are not relevant, but
they are still in the correct domain of words. This performance is most likely to improve by increasing the number of hidden layers and the number of epochs. However, given the limitations of the
simplistic example presented here, generation of these outputs clearly demonstrates the power of
RNN models.
11.6 Concluding Remarks
In this chapter, we introduce an important application of RNNs, namely the chatbot, which generates a response when a user enters a query in the form of a sentence in a natural language.
The characteristics of this application require a many-to-many RNN architecture, where the output sequence is generated by the RNN only after the user input sequence ends, and the input
and output sequences may differ in length and vary for each query. As described in Chapter 10,
a many-to-many RNN implemented as a chatbot typically employs hidden and embedding layers
to improve its prediction power and reduce the computation cost of the training process.
For this simple example, we show how to construct an RNN using the library functions available
in Python and Matlab. As its sample data, a small set of user text message exchanges taken from
the internet are used. The chatbot implementation demonstrates that it is possible to generate predictions that are not present in the sample data set when the user enters unique queries (using
only the words existing in sample training data). Even this simplistic RNN example is capable of
generating predictions that are bound to improve with more epochs and hidden layers, pointing to
the power of RNN-based models in implementing the chatbot class of applications.
11.7 Exercises for Chapter 11
(a)
(b)
Figure 11.6 Sample outputs generated by the Matlab script presented in Section 11.5: (a) user queries and
responses taken from the sample training data and the responses generated by the chatbot, and (b) unique
user queries and the responses generated by the chatbot.
11.7 Exercises for Chapter 11
1) Increase the number of epochs and observe its impact on training time and prediction power
(i.e. compare the responses to the same user queries).
2) Increase the number of hidden layers and observe its impact on training time and prediction
power (i.e. compare the responses to the same user queries).
3) Repeat Exercise 1 for batch size.
4) Repeat Exercise 1 for embedding layer size.
5) If you train the example RNN presented in Section 11.4 with a data set that has roughly twice
as many elements, what will be the impact on training cost and prediction power.
189
191
12
RNNs with Attention
12.1 Introduction
In this chapter we introduce attention, an enhancement mechanism for an RNN providing the
capability that the information generated by the previous hidden states is directly available
at each step [102, 103]. Recall from Chapter 10 that in an RNN, at step t, only the hidden state
information from the previous step ht−1 can be directly used. Recursively, ht−1 only has direct
access to the hidden state information of ht−2 from step (t − 2), which, in turn, has utilized the hidden state of ht−3 , and so on. But the indirect information from distant past hidden states only has a
weak influence on the current hidden state, especially as the distance to a past hidden state grows.
On the other hand, in an RNN with attention, denoted as RNN-Att, all hidden states generated at
the previous steps (i.e. ht−1 , ht−2 , · · · ht−k for a sequence of (k + 1) consecutive inputs) are directly
available when computing the hidden state ht and the output prediction ŷ t at step t. The main purpose of utilizing all previous hidden states at every step is to explore a possible relationship between
a current input and the previous hidden states generated by the inputs applied earlier. The attention
mechanism is especially important when processing input sequences where the relationship with
previously used inputs (typically more than one previous input) has a significant impact on how to
process the current input. Typical examples of such systems are machine translators, which convert sentences written (or spoken) in one natural language to another. We will introduce a simple
machine translation example using an RNN-Att in Chapter 13.
12.2 One-to-One RNN Cell with Attention
A one-to-one single cell RNN with attention is shown in Figure 12.1, which consists of three functional parts, namely an encoder, attention and decoder. Similar to the RNN presented in
Chapter 10, the encoder unit of an RNN-Att accepts an input vector xt at step t, which is scaled by
a weight matrix U and combined with both a bias vector b and the previous hidden state information ht−1 via a weight matrix W. After applying an activation function, which is selected as tanh
in Figure 12.1, hidden state ht is generated as the output of the encoder.
In the attention unit of an RNN-Att, current hidden state information ht from the encoder is
individually multiplied with each of the previous hidden states generated before this step (i.e.
∀h𝛿 where 𝛿 < t), and the results are individually fed into a softmax unit. These inputs are
called attention coefficients as shown in Figure 12.1. The outputs of the softmax unit
are called attention weights. Each output of softmax is then individually multiplied by
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
192
12 RNNs with Attention
Figure 12.1
A one-to-one single RNN cell with attention.
the corresponding attention weight (i.e. each hk is multiplied by the kth attention weight), which
are then summed to form the so-called context vector as the output of the attention unit.
We can see in Figure 12.1 that the RNN-Att decoder unit operates in the same way as in a regular
RNN, which scales its inputs (i.e. the context vector from the attention unit) by weight matrix
V and combines them with a bias vector c. The prediction of ŷ t is generated as the output of a
softmax unit.
After describing the operations of RNN-Att components, we can now see that an attention unit
attempts to directly utilize every single previous piece of hidden state information with the current input and current hidden state before generating a prediction. Recall in Chapter 10 that a
hidden state ht of a regular RNN only considers the immediate previous state of ht−1 , which in turn
includes its own immediate previous hidden state of ht−2 , and so on (see, for example, Eq. (10.3)
in Chapter 10). As the operations move forward step by step, the impact of past hidden states on
current computations gets weaker. However, in an RNN-Att, the attention unit includes every previous state information in generating its context vector, which is then used for output prediction in
the decoder.
12.3 Forward and Back Propagation
12.3 Forward and Back Propagation
Let us now explore the forward and back propagation operations in an RNN-Att. In Figure 12.2,
four steps of a one-to-one single RNN-Att cell are presented, where input Xt considers all hidden
states before it (i.e. ht−3 , · · · , ht−1 ) for computations in the attention unit. We can see in Figure 12.2
that when the input vector of xt is applied (i.e. the rightmost step), the attention unit accepts hidden
states of ht−3 , ht−2 , ht−1 and ht as it computes its context vector.
For the general case, when an input sequence of single characters x1 , x2 , · · · , x is applied to the
example RNN-Att shown in Figure 12.2, the hidden state of ht at step t is computed as
ht = Θh (U ⋅ Xt + W ⋅ ht−1 + b),
(12.1)
where the activation function Θh is typically selected as tanh. The inputs for the softmax unit
in attention are called attention scores. The number of attention scores grow as more inputs
are processed by an RNN-Att. Therefore, at step t, there are t inputs for the softmax unit, of which
the ith input is defined as
t,i = h†t ⋅ hi ,
Figure 12.2 Four steps of a one-to-one single RNN-Att cell, where input Xt considers all hidden states
before it (i.e. ht−3 , · · · , ht−1 ) for computations in attention unit.
193
194
12 RNNs with Attention
where † is the matrix transpose operation and (i = 1, · · · t). For example, at t = 1, the softmax
unit has only one input:
1,1 = h†1 ⋅ h1 .
However, at step t = 2, hidden state h2 is combined with both h1 and h2 before being fed into the
softmax unit:
2,1 = h†2 ⋅ h1
2,2 = h†2 ⋅ h2 .
Using Eq. (10.1) in Chapter 10, the output of the softmax unit in attention is
et,i
,
softmax(t,i ) = ∑t
t,j
j=1 e
(12.2)
where t is the current step. In attention, the outputs of the softmax unit are referred to as
attention weights, which are probabilities summing to 1. At the output of the softmax, the
ith attention weight is multiplied by the hidden state of hi . Then all of them are summed (row-wise
if multiple cells are present) to form a context vector, denoted as t at step t:
t =
t
∑
softmax(t,k ) ⋅ hk .
(12.3)
k=1
We can see from Eq. (12.3) that context vector t reflects the impact of previous hidden states
on a current hidden state value before being sent into the decoder, as shown in Figure 12.2. In the
decoder, the output before the softmax, Ωt , is
Ωt = V ⋅ t + c.
(12.4)
The output of the softmax unit in decoder, ŷ t , for each input, is
eΩt
softmax(Ωt ) = ŷ t = ∑||
for k = 1, … , ||.
eΩt,k
k=1
(12.5)
Finally, the cross-entropy loss for prediction ŷ t at step t is defined as
Lt = −yot ln( ŷ t ).
(12.6)
Once a forward propagation operation is performed, to reduce loss L, RNN-Att parameters such
as U, W, V, b and c must be updated using the partial derivatives of L with respect to these parameters. This update, which is conducted after each forward and back propagation during training,
ultimately minimizes L (or reduces it as much as it can), providing a more powerful prediction
mechanism. Steps of obtaining partial derivative formulas can be found in Appendix E. We strongly
recommend that the reader goes over these steps to fully understand the training process, which
resides at the heart of making successful predictions for all neural networks. The partial derivatives
of loss with respect to RNN-Att parameters are as follows:
𝜕Lt 𝜕 ŷ t
= ( ŷ t − yot )
𝜕 ŷ t 𝜕Ωt

𝜕L ∑
=
( ŷ t − yot ) ⋅ h†t
𝜕V
t=1
(12.7)
(12.8)
12.4 A Numerical Example

𝜕L ∑
=
( ŷ t − yot )
𝜕c
t=1

∑
𝜕L
=
( ŷ t − yot ) V †
𝜕W
t=1

𝜕L ∑
=
( ŷ t − yot ) V †
𝜕U
t=1

𝜕L ∑
=
( ŷ t − yot ) V †
𝜕b
t=1
(12.9)
( t m
∑∑
t,m
m=1 k=1
( t m
∑∑
m=1 k=1
†
(1 − h2j+1 )
)
)
(
2
(1 − hk ) hk−1
W
(12.10)
†
(1 − h2j+1 )
)
(
)
2
(1 − hk ) xk
(12.11)
†
(1 − h2j+1 )
)
(
)
2
(1 − hk ) .
(12.12)
j=k
∏
m−1
t,m
m=1 k=1
( t m
∑∑
∏
m−1
W
j=k
∏
m−1
t,m
W
j=k
Using the above partial derivatives, the parameters are updated by subtracting the gradient from
the current parameter value scaled by a learning rate of 𝜂:
𝜕L
𝜕V
𝜕L
cnew = c − 𝜂
𝜕c
𝜕L
Wnew = W − 𝜂
𝜕W
𝜕L
Unew = U − 𝜂
𝜕U
𝜕L
bnew = b − 𝜂 ,
𝜕b
Vnew = V − 𝜂
(12.13)
(12.14)
(12.15)
(12.16)
(12.17)
where subscript new indicates the updated value of a parameter on the left-hand side of each
equation.
12.4 A Numerical Example
We will use the same simplistic example presented in Chapter 10, where a user applies a single
character input using an alphabet of  = {h, e, l, o}, engaging one-hot encoding for inputs:
{
1 if i = c,
o
(12.18)
ci =
0 otherwise.
Table 12.1 shows all single character inputs and the expected outputs together with their
one-hot-encoded representations for each input in this example RNN-Att.
We will first examine the forward propagation process for four steps. Suppose the network parameters for this simple RNN-Att are initialized randomly as follows:
[
]
[
]
[ ]
U = 0.144 −0.232 −0.669 1.159 , W = −0.587 , b = 0
⎡0⎤
⎡−0.225⎤
⎢ ⎥
⎢
⎥
0
−0.305⎥
, c = ⎢ ⎥.
V =⎢
⎢0⎥
⎢ 0.491⎥
⎢0⎥
⎢ 0.277⎥
⎣ ⎦
⎣
⎦
195
196
12 RNNs with Attention
Table 12.1 Previous and current inputs and expected outputs with their one-hot forms
for the one-on-one RNN-Att example († denotes the matrix transpose operation).
Previous
character
Input
character
One-hot
encoding
-
h
h
e
l
Target
output
One-hot
encoding
x1 = [1 0 0 0]†
e
y1 = [0 1 0 0]†
e
†
x2 = [0 1 0 0]
l
y2 = [0 0 1 0]†
l
x3 = [0 0 1 0]†
l
y3 = [0 0 1 0]†
l
†
o
y4 = [0 0 0 1]†
x4 = [0 0 1 0]
Since all previous hidden state information are required for the attention mechanism, let us first
compute h1 , · · · , h4 at steps t = 1, · · · , 4 as follows:
ht = tanh(U ⋅ xt + W ⋅ ht−1 + b)
⎡1⎤
⎞
⎛
⎜[
]
] [ ] [ ]⎟ [
] ⎢0⎥ [
h1 = tanh ⎜ 0.144 −0.232 −0.669 1.159 ⋅ ⎢ ⎥ + −0.587 0 + 0 ⎟ = 0.143
⎢0⎥
⎟
⎜
⎢0⎥
⎟
⎜
⎣ ⎦
⎠
⎝
⎞
⎛
⎡0⎤
⎜[
]
][
] [ ]⎟ [
] ⎢1⎥ [
h2 = tanh ⎜ 0.144 −0.232 −0.669 1.159 ⋅ ⎢ ⎥ + −0.587 0.143 + 0 ⎟ = −0.306
⎟
⎜
⎢0⎥
⎟
⎜
⎢0⎥
⎣ ⎦
⎠
⎝
⎡0⎤
⎞
⎛
⎜[
]
] ⎢0⎥ [
][
] [ ]⎟ [
h3 = tanh ⎜ 0.144 −0.232 −0.669 1.159 ⋅ ⎢ ⎥ + −0.587 −0.306 + 0 ⎟ = −0.454
⎢1⎥
⎟
⎜
⎢0⎥
⎟
⎜
⎣ ⎦
⎠
⎝
⎡0⎤
⎞
⎛
⎜[
]
][
] [ ]⎟ [
] ⎢0⎥ [
h4 = tanh ⎜ 0.144 −0.232 −0.669 1.159 ⋅ ⎢ ⎥ + −0.587 −0.454 + 0 ⎟ = −0.382 .
⎢1⎥
⎟
⎜
⎢0⎥
⎟
⎜
⎣ ⎦
⎠
⎝
At step t = 1, for input x1 = h, the predicted output ŷ 1 is computed as follows:
1,1 = h†1 ⋅ h1 = [0.143] ⋅ [0.143] = [0.020]
1 = Θh (1,1 ) ⋅ h1 = softmax(1,1 ) ⋅ h1 = 1 [0.143]
⎡−0.225⎤
⎡0⎤ ⎡−0.032⎤
⎢
⎥
⎢ ⎥ ⎢
⎥
−0.305⎥
0
−0.044⎥
⋅ [0.143] + ⎢ ⎥ = ⎢
Ω1 = V ⋅ 1 + c = ⎢
⎢ 0.491⎥
⎢0⎥ ⎢ 0.070⎥
⎢ 0.277⎥
⎢0⎥ ⎢ 0.040⎥
⎣
⎦
⎣ ⎦ ⎣
⎦
⎡0.240⎤
⎢
⎥
0.237⎥
.
ŷ 1 = Θŷ (Ω1 ) = softmax(Ω1 ) = ⎢
⎢0.266⎥
⎢0.258⎥
⎣
⎦
12.4 A Numerical Example
Based on the softmax outputs, the prediction is ŷ 1 = l (i.e. the output with the highest value);
however, the expected output from Table 12.1 is yo1 = e. Loss L1 is
⎡0⎤ ⎡0.240⎤
⎢ ⎥ ⎢
⎥
1
0.237⎥
= 1.440.
L1 = −yo1 ln( ŷ 1 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.266⎥
⎢0⎥ ⎢0.258⎥
⎣ ⎦ ⎣
⎦
At step t = 2, for input x2 = e, computation of prediction ŷ 2 proceeds as follows:
2,1 = h†2 ⋅ h1 = [−0.306] ⋅ [0.143] = [−0.044]
2,2 = h†2 ⋅ h2 = [−0.306] ⋅ [−0.306] = [0.094]
2 =
2
∑
Θ (2,k ) ⋅ hk = [0.466] ⋅ [0.143] + [0.534] ⋅ [−0.306] = [−0.097]
k=1
⎡0⎤ ⎡ 0.022⎤
⎡−0.225⎤
⎥
⎢ ⎥ ⎢
⎢
⎥
0.029⎥
0
−0.305⎥
[−0.097] + ⎢ ⎥ = ⎢
Ω2 = V ⋅ 2 + c = ⎢
⎢0⎥ ⎢−0.047⎥
⎢ 0.490⎥
⎢0⎥ ⎢−0.027⎥
⎢ 0.277⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
⎡0.257⎤
⎢
⎥
0.259⎥
ŷ 2 = Θŷ = softmax(Ω2 ) = ⎢
.
⎢0.240⎥
⎢0.245⎥
⎣
⎦
Based on the softmax unit outputs, for input x2 = e, our simple RNN-Att generates a predicted
output as ŷ 2 = e, whereas the expected output from Table 12.1 is yo2 = l. For this step, the loss is
computed as
⎡0⎤ ⎡0.257⎤
⎥
⎢ ⎥ ⎢
0.259⎥
0
= 1.428.
L2 = −yo2 ln( ŷ 2 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.240⎥
⎢0⎥ ⎢0.245⎥
⎦
⎣ ⎦ ⎣
At step t = 3, the predicted output ŷ 3 for input x3 = l is computed as follows:
3,1 = h†3 ⋅ h1 = [−0.454] [0.143] = [−0.065]
3,2 = h†3 ⋅ h2 = [−0.454] [−0.306] = [0.139]
3,3 = h†3 ⋅ h3 = [−0.454] [−0.454] = [0.206]
3 =
3
∑
Θ (3,k ) hk = [0.283] [0.143] + [0.347] [−0.306] + [0.371] [−0.454] = [−0.234]
k=1
⎡0⎤ ⎡ 0.053⎤
⎡−0.225⎤
⎥
⎢ ⎥ ⎢
⎢
⎥
0.071⎥
0
−0.305⎥
⋅ [−0.234] + ⎢ ⎥ = ⎢
Ω3 = V ⋅ 3 + c = ⎢
⎢0⎥ ⎢−0.115⎥
⎢ 0.491⎥
⎢0⎥ ⎢−0.065⎥
⎢ 0.277⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
197
198
12 RNNs with Attention
⎡0.266⎤
⎢
⎥
0.271⎥
ŷ 3 = Θŷ (Ω3 ) = softmax(Ω3 ) = ⎢
.
⎢0.225⎥
⎢0.237⎥
⎣
⎦
Based on the highest value of the softmax unit output, for input x3 = l, our model generates a
prediction of ŷ 3 = e, as opposed to the expected output of yo3 = l from Table 12.1. For step t = 3,
the loss is
⎡0⎤ ⎡0.266⎤
⎥
⎢ ⎥ ⎢
0.271⎥
0
= 1.490.
L3 = −yo3 ln( ŷ 3 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.225⎥
⎢0⎥ ⎢0.237⎥
⎦
⎣ ⎦ ⎣
Similarly, at step t = 4 of forward propagation in training, for input x4 = l, the output prediction
ŷ 4 is computed as
4,1 = h†4 ⋅ h1 = [−0.382] [0.143] = [−0.055]
4,2 = h†4 ⋅ h2 = [−0.382] [−0.306] = [0.117]
4,3 = h†4 ⋅ h3 = [−0.382] [−0.454] = [0.173]
4,3 = h†4 ⋅ h4 = [−0.382] [−0.382] = [0.146]
4 =
4
∑
Θ (4,k ) hk
k=1
= [0.214] [0.143] + [0.254] [−0.306] + [0.269] [−0.454] + [0.262] [−0.382]
= [−0.269]
⎡0⎤ ⎡ 0.060 ⎤
⎡−0.225⎤
⎥
⎢ ⎥ ⎢
⎢
⎥
−0.305
⎥ ⋅ [−0.269] + ⎢0⎥ = ⎢ 0.082 ⎥
Ω4 = V ⋅ 4 + c = ⎢
⎢0⎥ ⎢−0.1321⎥
⎢ 0.491⎥
⎢0⎥ ⎢ −0.075 ⎥
⎢ 0.277⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
⎡0.269⎤
⎢
⎥
0.275⎥
ŷ 4 = Θŷ (Ω4 ) = softmax(Ω4 ) = ⎢
.
⎢0.222⎥
⎢0.235⎥
⎣
⎦
From the softmax unit output, for applying input x4 = l a second time, our model generates as
ŷ 4 = e, instead of yo4 = o. The loss at step t = 4 is computed as
⎡0⎤ ⎡0.269⎤
⎥
⎢ ⎥ ⎢
0.275⎥
0
= 1.449.
L4 = −yo4 ln( ŷ 4 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.222⎥
⎢1⎥ ⎢0.235⎥
⎦
⎣ ⎦ ⎣
The total loss for  = 4 steps then becomes
L=

∑
t=1
Lt = 1.440 + 1.428 + 1.490 + 1.449 = 5.807.
12.4 A Numerical Example
Let us now focus on the back propagation process using the partial derivatives of total loss with
respect to model parameters. From Eq. (12.7), we have
⎡0.240⎤ ⎡0⎤ ⎡ 0.240⎤
⎢
⎥ ⎢ ⎥ ⎢
⎥
𝜕L1 𝜕 ŷ 1
0.237⎥ ⎢1⎥ ⎢−0.763⎥
= ( ŷ 1 − yo1 ) = ⎢
−
=
⎢0.266⎥ ⎢0⎥ ⎢ 0.266⎥
𝜕 ŷ 1 𝜕Ω1
⎢0.258⎥ ⎢0⎥ ⎢ 0.258⎥
⎣
⎦ ⎣ ⎦ ⎣
⎦
⎡0.257⎤ ⎡0⎤ ⎡ 0.257⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L2 𝜕 ŷ 2
0.259⎥ ⎢0⎥ ⎢ 0.259⎥
=
−
= ( ŷ 2 − yo2 ) = ⎢
⎢0.240⎥ ⎢1⎥ ⎢−0.760⎥
𝜕 ŷ 2 𝜕Ω2
⎢0.245⎥ ⎢0⎥ ⎢ 0.245⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
⎡0.266⎤ ⎡0⎤ ⎡ 0.266⎤
⎢
⎥ ⎢ ⎥ ⎢
⎥
̂
𝜕L3 𝜕 y3
0.271⎥ ⎢0⎥ ⎢ 0.271⎥
= ( ŷ 3 − yo3 ) = ⎢
−
=
⎢0.225⎥ ⎢1⎥ ⎢−0.775⎥
𝜕 ŷ 3 𝜕Ω3
⎢0.237⎥ ⎢0⎥ ⎢ 0.237⎥
⎣
⎦ ⎣ ⎦ ⎣
⎦
⎡0.269⎤ ⎡0⎤ ⎡ 0.269⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L4 𝜕 ŷ 4
0.275⎥ ⎢0⎥ ⎢ 0.275⎥
.
=
−
= ( ŷ 4 − yo4 ) = ⎢
⎢0.222⎥ ⎢0⎥ ⎢ 0.222⎥
𝜕 ŷ 4 𝜕Ω4
⎢0.235⎥ ⎢1⎥ ⎢−0.765⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
Partial derivatives of L with respect to V, c, W, U and b are computed using Eqs. (12.8), (12.9),
(12.10), (12.11), and (12.12), respectively, as follows:
⎡−0.268⎤
⎢
⎥
𝜕L ⎢−0.417⎥
=
𝜕V ⎢ 0.537⎥
⎢ 0.147⎥
⎣
⎦
⎡ 1.032⎤
⎢
⎥
𝜕L ⎢ 0.042⎥
=
𝜕c ⎢−1.048⎥
⎢−0.026⎥
⎣
⎦
[
]
𝜕L
= 0.0274
𝜕W
]
]
𝜕L [
𝜕L [
= 0.1817 −0.3287 −0.2169 0
= −0.364 .
𝜕U
𝜕b
Using a learning rate of 𝜂 = 0.1 (an arbitrarily chosen value), we update parameters of V, c, W,
U and b using Eqs. (12.13), (12.14), (12.15), (12.16) and (12.17), respectively, as follows:
⎡−0.268⎤ ⎡−0.222⎤
⎡−0.225⎤
⎥
⎥ ⎢
⎢
⎥
⎢
−0.417⎥ ⎢−0.301⎥
𝜕L ⎢−0.305⎥
=
− 0.1 ⎢
=
Vnew = V − 𝜂
⎢ 0.537⎥ ⎢ 0.485⎥
𝜕V ⎢ 0.491⎥
⎢ 0.147⎥ ⎢ 0.275⎥
⎢ 0.277⎥
⎦
⎦ ⎣
⎣
⎦
⎣
⎡ 1.032⎤ ⎡−0.0103⎤
⎡0⎤
⎥
⎢
⎥ ⎢
⎢ ⎥
0.042⎥ ⎢−0.0004⎥
𝜕L ⎢0⎥
cnew = c − 𝜂
=
− 0.1 ⎢
=
⎢−1.048⎥ ⎢ 0.0105⎥
𝜕c ⎢0⎥
⎢−0.026⎥ ⎢ 0.0003⎥
⎢0⎥
⎦
⎣
⎦ ⎣
⎣ ⎦
Wnew = W − 𝜂
[
]
[
] [
]
𝜕L
= −0.5870 − 0.1 ⋅ 0.0274 = −0.5872
𝜕W
𝜕L
Unew = U − 𝜂
𝜕U
[
]
[
]
= 0.144 −0.232 −0.669 1.159 − 0.1 0.182 −0.329 −0.217 0.0
199
200
12 RNNs with Attention
[
]
= 0.142 −0.228 −0.667 1.159
[
] [
]
𝜕L [ ]
= 0 − 0.1 −0.364 = 0.0036 ,
𝜕b
where subscript new is used on the left-hand sides of these equations to denote the updated values
of the parameters, whereas the values on the right-hand side correspond to the values before the
update. Using the updated parameter values, we continue with another forward propagation with
four steps as above, which yields the following loss values (computations are not shown):
bnew = b − 𝜂
⎡0⎤ ⎡ 0.237 ⎤
⎥
⎢ ⎥ ⎢
⋅0.239⎥
1
= 1.440
L1 = −yo1 ln( ŷ 1 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢ 0.268 ⎥
⎢0⎥ ⎢ 0.258 ⎥
⎦
⎣ ⎦ ⎣
⎡0⎤ ⎡0.254⎤
⎥
⎢ ⎥ ⎢
0.258⎥
0
= 1.416
L2 = −yo2 ln( ŷ 2 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.243⎥
⎢0⎥ ⎢0.245⎥
⎦
⎣ ⎦ ⎣
⎡0⎤ ⎡0.2633⎤
⎥
⎢ ⎥ ⎢
0.271 ⎥
0
= 1.476
L3 = −yo3 ln( ŷ 3 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢ 0.229 ⎥
⎢0⎥ ⎢ 0.237 ⎥
⎦
⎣ ⎦ ⎣
⎡0⎤ ⎡0.266⎤
⎥
⎢ ⎥ ⎢
0.274⎥
0
= 1.447,
L4 = −yo4 ln( ŷ 4 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.225⎥
⎢1⎥ ⎢0.235⎥
⎦
⎣ ⎦ ⎣
where the predictions for the second forward propagation are shown as boldface characters (i.e.
the inputs of x1 = h, x2 = e, x3 = l, and x4 = l generate the predictions of ŷ 1 = l, ŷ 2 = e, ŷ 3 = e,
and ŷ 4 = e, respectively). The total loss L for four steps is
L=

∑
Lt = 1.440 + 1.416 + 1.476 + 1.447 = 5.779,
t=1
which was 5.807 for the first epoch as shown above. As training continues with more epochs, the
total error value is expected to reduce. For this example, the error values are found as 4.679 and
2.104 for 100 and 1000 epochs, respectively.
12.5 Embedding Layer
As described in Section 10.5 in Chapter 10, embedding is used to lower the dimensions of input
vectors during training and, hence, reduce the computational cost of training for real-life applications typically engaging hundreds of thousands of samples. In Figure 12.3, we show inclusion of
embedding in a one-to-one single RNN-Att cell, where an embedding matrix E is multiplied with a
one-hot encoded input vector before it is applied to the encoder. The dimensions of the E matrix is
(e × ||), where e and || are the embedding width and the dictionary size, respectively. Application
12.5 Embedding Layer
Figure 12.3
A one-to-one single RNN-Att cell with an embedding layer.
of the E matrix to the one-hot-encoded input elements reduces the dimensions of the input vector
from (|| × 1) to (e × 1) as follows:
xt = E ⋅ xto ,
(12.19)
where † represents the matrix transpose operation. After embedding weight matrix E is applied to
the one-hot-coded input vector, the forward propagation process described by Eqs. (12.1)–(12.12)
given in Section 12.2 remains the same for this case. Similarly, the back propagation operations and
partial derivations of L with respect to RNN-Att parameters given in Eqs. (12.7)–(12.12) are valid
after adding an embedding layer. However, since there is now an additional parameter called E, we
need to compute the partial derivation of L with respect to E as follows (step-by-step description of
the derivation of the formulas is presented in Appendix E):
)
( t m

m−1
∑∑
∏
)
(
𝜕L ∑
o
†
†
2
2
o
.
(12.20)
=
( ŷ t − yt )V
t,m
W (1 − hj+1 ) (1 − hk )Uxk
𝜕E
t=1
m=1 k=1
j=k
Eqs. (12.13)–(12.17) given in Section 12.2 show how to update the RNN-Att parameters for a
learning rate of 𝜂 after a back propagation operation is completed. When an embedding layer is
employed, a similar equation for updating E will be needed as follows:
Enew = E − 𝜂
𝜕L
.
𝜕E
(12.21)
201
202
12 RNNs with Attention
Figure 12.4 Four steps of a one-to-one single RNN-Att cell using an embedding layer, where the
dimension of a one-hot encoded input vector Xto is reduced from (|| × 1) to (e × 1) for an alphabet size of
|| and embedding size of e.
12.6 A Numerical Example with Embedding
Let us now examine the training process for one epoch in a one-to-one single cell RNN-Att that
employs an embedding layer at its inputs (Figure 12.4). Consider the same example given in
Table 12.1 in Section 12.4, where four characters are applied at four steps. Suppose the weight
matrices for the RNN-Att are randomly initialized as follows:
[
]
[
]
[
]
0.818 −0.360 0.127 −0.036
E=
U = 0.144 −0.2315
W = −0.535
0.461 −0.482 0.154 −0.568
[
]†
[ ]
[
]†
b= 0
V = 0.695 − 0.440 − 0.225 − 0.305
c= 0000 .
At steps t = 1, · · · , 4, we have the outputs of embedding layer from Eq. (12.19) and the corresponding hidden states from Eq. (12.1) as follows:
⎡1⎤
]
[
] ⎢ ⎥ [
0.818
0.818 −0.360 0.127 −0.036 ⎢0⎥
=
⋅
x1 =
0.461
0.461 −0.482 0.154 −0.568 ⎢0⎥
⎢0⎥
⎣ ⎦
12.6 A Numerical Example with Embedding
⎡0⎤
[
] ⎢ ⎥ [
]
0.818 −0.360 0.127 −0.036 ⎢1⎥
−0.360
x2 =
⋅
=
0.461 −0.482 0.154 −0.568 ⎢0⎥
−0.482
⎢0⎥
⎣ ⎦
⎡0⎤
]
[
] ⎢ ⎥ [
0.127
0.818 −0.360 0.127 −0.036 ⎢0⎥
=
x3 =
⋅
0.154
0.461 −0.482 0.154 −0.568 ⎢1⎥
⎢0⎥
⎣ ⎦
⎡0⎤
]
[
] ⎢ ⎥ [
0.127
0.818 −0.360 0.127 −0.036 ⎢0⎥
=
x4 =
⋅
0.154
0.461 −0.482 0.154 −0.568 ⎢1⎥
⎢0⎥
⎣ ⎦
(
h1 = tanh
(
h2 = tanh
(
h3 = tanh
(
h4 = tanh
[
[
]
)
] 0.818
[
][ ] [ ]
[
]
+ −0.535 0 + 0 = 0.011
0.144 −0.232 ⋅
0.461
[
[
]
)
] −0.360
[
][
] [ ]
[
]
+ −0.535 0.011 + 0 = 0.054
0.144 −0.232
−0.482
[
[
]
)
] 0.127
[
][
] [ ]
[
]
+ −0.535 0.054 + 0 = −0.046
0.144 −0.232
0.154
[
[
]
)
] 0.127
[
][
] [ ]
[
]
+ −0.535 −0.046 + 0 = 0.007 .
0.144 −0.232
0.154
At step t = 1, computation of predicted output ŷ 1 proceeds as follows:
1,1 , = h†1 ⋅ h1 = [0.011] ⋅ [0.011] = [0.0001]
1 = Θ (1,1 ) ⋅ h1 = softmax(1,1 ) ⋅ h1 = [0.011]
⎡0⎤ ⎡ 0.008⎤
⎡ 0.695⎤
⎥
⎢ ⎥ ⎢
⎢
⎥
−0.005⎥
0
−0.440⎥
⋅ [0.011] + ⎢ ⎥ = ⎢
Ω1 = V ⋅ 1 + c = ⎢
⎢0⎥ ⎢−0.003⎥
⎢−0.225⎥
⎢0⎥ ⎢−0.003⎥
⎢−0.305⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
⎡0.252⎤
⎢
⎥
0.249⎥
ŷ 1 = Θŷ (Ω1 ) = softmax(Ω1 ) = ⎢
.
⎢0.250⎥
⎢0.249⎥
⎣
⎦
For input x1 = h, our model generates a prediction of ŷ 1 = h instead of the expected output of
yo1 = e. The loss is
⎡0⎤ ⎡0.252⎤
⎥
⎢ ⎥ ⎢
0.249⎥
1
= 1.390.
L1 = −yo1 ln( ŷ 1 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.250⎥
⎢0⎥ ⎢0.249⎥
⎦
⎣ ⎦ ⎣
203
204
12 RNNs with Attention
At step t = 2, for input x1 = e , the predicted output ŷ 2 is computed as follows:
2,1 = h†2 ⋅ h1 = [0.054] ⋅ [0.011] = [0.001]
2 =
2,2 = h†2 ⋅ h2 = [0.054] ⋅ [0.054] = [0.003]
2
∑
Θ (2,k ) ⋅ hk = [0.500] ⋅ [0.011] + [0.500] ⋅ [0.054] = [0.033]
k=1
⎡0⎤ ⎡ 0.023⎤
⎡ 0.695⎤
⎥
⎢ ⎥ ⎢
⎢
⎥
−0.440
⎥ ⋅ [0.033] + ⎢0⎥ = ⎢−0.014⎥
Ω2 = V ⋅ 2 + c = ⎢
⎢0⎥ ⎢−0.007⎥
⎢−0.225⎥
⎢0⎥ ⎢−0.010⎥
⎢−0.305⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
⎡0.256⎤
⎢
⎥
0.247⎥
ŷ 2 = Θŷ (Ω2 ) = softmax(Ω2 ) = ⎢
.
⎢0.249⎥
⎢0.248⎥
⎣
⎦
For input x2 = e, the prediction generated by our model is ŷ 2 = h instead of yo2 = l. The loss is
⎡0⎤ ⎡0.256⎤
⎥
⎢ ⎥ ⎢
0.247⎥
0
= 1.392.
L2 = −yo2 ln( ŷ 2 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.249⎥
⎢0⎥ ⎢0.248⎥
⎦
⎣ ⎦ ⎣
At step t = 3, the user applies input x3 = l, for which a predicted output ŷ 3 is computed as follows:
3,1 = [−0.046] ⋅ [0.011] = [−0.0005]
3,2 = [−0.046] ⋅ [0.054] = [−0.002]
3,3 = [−0.046] ⋅ [−0.046] = [0.002]
3 =
3
∑
Θ (3,k ) ⋅ hk
k=1
= [0.333] ⋅ [0.011] + [0.333] ⋅ [0.054] + [0.334] ⋅ [−0.046] = [0.006]
⎡0⎤ ⎡ 0.004⎤
⎡ 0.695⎤
⎥
⎢ ⎥ ⎢
⎢
⎥
−0.003⎥
0
−0.440⎥
⋅ [0.006] + ⎢ ⎥ = ⎢
Ω3 = V ⋅ 3 + c = ⎢
⎢0⎥ ⎢−0.001⎥
⎢−0.225⎥
⎢0⎥ ⎢−0.002⎥
⎢−0.305⎥
⎦
⎣ ⎦ ⎣
⎣
⎦
⎡0.251⎤
⎢
⎥
0.249⎥
ŷ 3 = Θŷ (Ω3 ) = softmax(Ω3 ) = ⎢
.
⎢0.250⎥
⎢0.250⎥
⎣
⎦
For input x3 = l, the prediction is ŷ 3 = h instead of yo3 = l. The loss L3 is
⎡0⎤ ⎡0.251⎤
⎥
⎢ ⎥ ⎢
0.249⎥
0
= 1.387.
L3 = −yo3 ln( ŷ 3 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.250⎥
⎢0⎥ ⎢0.250⎥
⎦
⎣ ⎦ ⎣
12.6 A Numerical Example with Embedding
Similarly, at step t = 4, for input is x4 = l, the expected output ŷ 4 is computed as follows:
4,1 = [0.007] ⋅ [0.011] = [0.0001]
4,2 = [0.007] ⋅ [0.054] = [0.0004]
4,3 = [0.007] ⋅ [−0.046] = [−0.0003]
4,3 = [0.007] ⋅ [0.007] = [0.0001]
4 =
4
∑
Θ (4,k ) ⋅ hk
k=1
4 = [0.250] ⋅ [0.011] + [0.250] ⋅ [0.054] + [0.250] ⋅ [−0.046] + [0.250] ⋅ [0.007] = [0.007]
⎡ 0.695⎤
⎡0⎤ ⎡ 0.005⎤
⎢
⎥
⎢ ⎥ ⎢
⎥
−0.440
⎥ ⋅ [−0.269] + ⎢0⎥ = ⎢−0.003⎥
Ω4 = V ⋅ 4 + c = ⎢
⎢−0.225⎥
⎢0⎥ ⎢−0.002⎥
⎢−0.305⎥
⎢0⎥ ⎢−0.002⎥
⎣
⎦
⎣ ⎦ ⎣
⎦
⎡0.251⎤
⎢
⎥
0.249⎥
.
ŷ 4 = Θŷ (Ω4 ) = softmax(Ω4 ) = ⎢
⎢0.250⎥
⎢0.250⎥
⎣
⎦
Based on the highest value out of the softmax unit, the prediction is ŷ 4 = h instead of yo4 = o.
For this prediction, the loss is
⎡0⎤ ⎡0.251⎤
⎥
⎢ ⎥ ⎢
0.249⎥
0
= 1.388.
L4 = −yo4 ln( ŷ 4 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.250⎥
⎢1⎥ ⎢0.250⎥
⎦
⎣ ⎦ ⎣
The total loss for  = 4 steps is
L=

∑
Lt = 1.390 + 1.392 + 1.387 + 1.388 = 5.557.
t=1
After the forward propagation is complete, let us now consider operations for a back propagation.
From Eq. (12.7), we have
⎡0.252⎤ ⎡0⎤ ⎡ 0.252⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L1 𝜕 ŷ 1
0.249⎥ ⎢1⎥ ⎢−0.751⎥
=
−
= ( ŷ 1 − yo1 ) = ⎢
⎢0.250⎥ ⎢0⎥ ⎢ 0.250⎥
𝜕 ŷ 1 𝜕Ω1
⎢0.249⎥ ⎢0⎥ ⎢ 0.249⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
⎡0.256⎤ ⎡0⎤ ⎡ 0.256⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L2 𝜕 ŷ 2
0.247⎥ ⎢0⎥ ⎢ 0.247⎥
=
−
= ( ŷ 2 − yo2 ) = ⎢
⎢0.249⎥ ⎢1⎥ ⎢−0.751⎥
𝜕 ŷ 2 𝜕Ω2
⎢0.248⎥ ⎢0⎥ ⎢ 0.248⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
⎡0.251⎤ ⎡0⎤ ⎡ 0.251⎤
⎥
⎢
⎥ ⎢ ⎥ ⎢
𝜕L3 𝜕 ŷ 3
0.249⎥ ⎢0⎥ ⎢ 0.249⎥
=
−
= ( ŷ 3 − yo3 ) = ⎢
⎢0.250⎥ ⎢1⎥ ⎢−0.750⎥
𝜕 ŷ 3 𝜕Ω3
⎢0.250⎥ ⎢0⎥ ⎢ 0.250⎥
⎦
⎣
⎦ ⎣ ⎦ ⎣
205
206
12 RNNs with Attention
⎡0.251⎤ ⎡0⎤ ⎡ 0.251⎤
⎢
⎥ ⎢ ⎥ ⎢
⎥
𝜕L4 𝜕 ŷ 4
0.249⎥ ⎢0⎥ ⎢ 0.249⎥
= ( ŷ 4 − yo4 ) = ⎢
−
=
.
⎢0.250⎥ ⎢0⎥ ⎢ 0.250⎥
𝜕 ŷ 4 𝜕Ω4
⎢0.250⎥ ⎢1⎥ ⎢−0.750⎥
⎣
⎦ ⎣ ⎦ ⎣
⎦
Let us now compute the partial derivatives of L with respect to V, c, W, U, b and E using
Eqs. (12.8), (12.9), (12.10), (12.11), (12.12) and (E.25), respectively, as follows:
⎡−0.268⎤
⎢
⎥
𝜕L ⎢−0.417⎥
=
𝜕V ⎢ 0.537⎥
⎢ 0.147⎥
⎣
⎦
⎡ 1.011⎤
⎢
⎥
𝜕L ⎢−0.005⎥
=
𝜕c ⎢−1.002⎥
⎢−0.003⎥
⎣
⎦
]
𝜕L [
= 0.361 0.174
𝜕U
]
𝜕L [
= 0.776
𝜕b
[
]
𝜕L
= 0.003
𝜕W
[
]
0.054 0.011 0.038 0.0
𝜕L
=
.
−0.087 −0.017 −0.060 0.0
𝜕E
Using an arbitrarily chosen learning rate of 𝜂 = 0.1, updated values for V, c, W, U, b and E are
computed using Eqs. (12.13), (12.14), (12.15), (12.16), (12.17) and (12.21), respectively, as follows:
⎡ 0.007⎤ ⎡ 0.694⎤
⎡ 0.695⎤
⎥
⎢
⎥ ⎢
⎢
⎥
−0.005⎥ ⎢−0.440⎥
𝜕L ⎢−0.440⎥
=
− 0.1 ⎢
=
Vnew = V − 𝜂
⎢−0.001⎥ ⎢−0.224⎥
𝜕V ⎢−0.225⎥
⎢−0.001⎥ ⎢−0.305⎥
⎢−0.305⎥
⎦
⎣
⎦ ⎣
⎣
⎦
⎡ 1.011⎤ ⎡−0.1011⎤
⎡0⎤
⎥
⎢
⎥ ⎢
⎢ ⎥
−0.005⎥ ⎢ 0.0005⎥
𝜕L ⎢0⎥
cnew = c − 𝜂
=
− 0.1 ⎢
=
⎢−1.002⎥ ⎢ 0.1002⎥
𝜕c ⎢0⎥
⎢−0.003⎥ ⎢ 0.0003⎥
⎢0⎥
⎦
⎣
⎦ ⎣
⎣ ⎦
Wnew = W − 𝜂
Unew = U − 𝜂
bnew = b − 𝜂
[
]
[
] [
]
𝜕L
= −0.535 − 0.1 0.003 = −0.536
𝜕W
]
[
] [
]
𝜕L [
= 0.144 −0.232 − 0.1 0.361 0.174 = 0.108 − 0.249
𝜕U
[
] [
]
𝜕L [ ]
= 0 − 0.1 0.776 = −0.078
𝜕b
𝜕L
Enew = E − 𝜂
𝜕E
[
]
[
]
0.818 −0.360 0.127 −0.036
0.054 0.011 0.038 0.0
=
− 0.1
0.461 −0.482 0.154 −0.568
−0.087 −0.017 −0.060 0.0
[
]
0.812 −0.361 0.123 −0.036
=
,
0.470 −0.481 0.160 −0.568
where subscript new denotes the updated parameter value. We then perform another forward propagation operation with four steps, which yields the following loss values (computations are not
shown):
⎡0⎤ ⎡0.207⎤
⎥
⎢ ⎥ ⎢
0.259⎥
1
= 1.352
L1 = −yo1 ln( ŷ 1 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.279⎥
⎢0⎥ ⎢0.255⎥
⎦
⎣ ⎦ ⎣
12.8 Exercises for Chapter 12
⎡0⎤ ⎡0.221⎤
⎢ ⎥ ⎢
⎥
0
0.252⎥
L2 = −yo2 ln( ŷ 2 ) = − ⎢ ⎥ ln ⎢
= 1.286
⎢1⎥ ⎢0.277⎥
⎢0⎥ ⎢0.251⎥
⎣ ⎦ ⎣
⎦
⎡0⎤ ⎡0.215⎤
⎥
⎢ ⎥ ⎢
0.255⎥
0
= 1.281
L3 = −yo3 ln( ŷ 3 ) = − ⎢ ⎥ ln ⎢
⎢1⎥ ⎢0.278⎥
⎢0⎥ ⎢0.253⎥
⎦
⎣ ⎦ ⎣
⎡0⎤ ⎡0.216⎤
⎥
⎢ ⎥ ⎢
0.254⎥
0
= 1.377,
L4 = −yo4 ln( ŷ 4 ) = − ⎢ ⎥ ln ⎢
⎢0⎥ ⎢0.278⎥
⎢1⎥ ⎢0.252⎥
⎦
⎣ ⎦ ⎣
where for inputs x1 = h, x2 = e, x3 = l and x4 = l, the predictions are ŷ 1 = l, ŷ 2 = l, ŷ 3 = l and
ŷ 4 = l, respectively (shown as boldface characters in the loss equations above). The total loss L for
 = 4 steps is
L=

∑
Lt = 1.352 + 1.287 + 1.281 + 1.377 = 5.297,
t=1
which was 5.557 for the first epoch as shown above. As training continues with more epochs, the
total error value is expected to reduce. For this example, the error values were found as 2.089 and
1.926 for 100 and 1000 epochs, respectively.
12.7 Concluding Remarks
In this chapter we introduced an attention mechanism, where all previous hidden states are directly
used at each step during training. This is especially important for applications such as machine
translation, where the current output must use previously applied inputs, whereas in an ordinary
RNN the main focus for the current input is the current and the immediate previous hidden states.
Forward and back propagation operations, computation of loss function and updating the parameters were explained in detail for a simple one-to-one single cell RNN-Att.
An RNN with attention is a basic building block of several more advanced language models,
including LSTM and transformers [ref]. We expect that the reader will be able to understand these
advanced models without difficulty after digesting the materials presented in this book.
12.8 Exercises for Chapter 12
1) What is the impact of increasing the learning rate 𝜂 of an RNN-Att on its prediction power?
Derive an approximate formula for it. Compare your answer with Exercise 1 in Chapter 10.
2) How does increasing the number of epochs affect the prediction power of an RNN-Att? Derive
an approximate formula for it and compare it with Exercise 3 in Chapter 10.
3) Derive the dimensions of the weight matrices for a three-cell RNN-Att using a single hidden
layer.
207
208
12 RNNs with Attention
4) Derive an order-of-magnitude approximation formula for the computational cost of one
epoch for an RNN-Att with and without embedding layers. Keep all other parameters fixed to
demonstrate the impact of embedding layers.
5) Derive the dimensions of the weight matrices for a three-cell RNN-Att using two hidden layers
(use the RNN given in Figure 12.3 as a guide).
209
13
RNNs with Attention for Machine Translation
13.1 Introduction
In this chapter we introduce a simple example for machine translation using an RNN with
the attention mechanism. As discussed in Chapter 12, for applications where it is important to
remember the hidden states generated by previous words in translation, we need the attention
mechanism.
In natural languages, placements of the subject, verb, preposition and tense information in
a sentence significantly vary from one language to another. For example, if one would like
˛dam tylko ja.
to say The sky is viewed only by me in Polish, it would be Niebo ogla
However, the statement The sky is viewed only by you is translated into Polish as Niebo
˛dasz tylko ty. As can be seen in Figures 13.1a and b, the translation of this sentence
ogla
into Polish cannot be determined until the user enters the last English word either as me in the
first sentence or you in the second. Entering the last word as me causes the inclusion of words
˛dam and ja in the Polish translation, whereas the last word of you will place the words
ogla
˛dasz and ty in the Polish sentence.
ogla
Another observation from Figure 13.1 is the difference between the number of words in the
English input and Polish output sentences. As described in Section 11.4 in Chapter 11, one popular solution for handling input and output sequences with different lengths is adopting padding
techniques, which insert blank characters for shorter sequences to make all inputs and outputs
appear equal in length.
This example clearly illustrates that, in applications such as machine translation, the influence of
a word at the end of an input sequence has the potential to completely change the predicted output,
which necessitates keeping previous hidden states available at every step. Also in translations an
input sequence may generate an output sequence with a different length. Such characteristics indicate that for a class of applications like machine translation, we need to employ a many-to-many
RNN-Att. Because of the large number of samples, each with a large dictionary, RNN-Att architectures with multiple hidden and embedding layers are customary.
In this chapter, we present a machine translation example where a simplistic many-to-many
RNN-Att with hidden and embedding layers is implemented for translating sentences from German
to English. Due to space restrictions, we use a small set of 3000 translation samples with a total of
1576 German and 896 English words. Python and Matlab implementations for the translator are
presented in Sections 13.3 and 13.4, respectively.
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
210
13 RNNs with Attention for Machine Translation
(a)
(b)
Figure 13.1 English to Polish translation example, where the Polish sentence is not determined until the
last word in English is entered.
13.2 Many-to-Many Architecture
For a class of applications such as machine translation, an RNN is expected to remember the previous inputs until the entire sequence is entered. Also, since even the last words in an input sequence
may affect the translation, generation of outputs does not start until the entire input sequence is
entered. Therefore, an attention mechanism with a many-to-many structure is needed in the RNN
architecture. Figure 13.2 shows a many-to-many RNN-Att with multiple hidden layers used for a
machine translator, where no outputs are generated until the user completes entering an input
sequence in step k, after which external outputs are generated and fed back into the RNN-Att as
inputs. As can be seen in Figure 13.2, the outputs Ŷ k , Ŷ k+1 , … are applied to the RNN-Att as inputs
Xk+1 , Xk+2 , … , respectively.
Since machine translators require a large set of sample data for training and use dictionaries
with high numbers of words, it is necessary to use multiple embedding layers to reduce the input
Figure 13.2 A many-to-many RNN-Att with multiple hidden layers used for a machine translator
implementation, where output generation is delayed until step k, after which external outputs are
generated and fed back into the RNN-Att as inputs (i.e. outputs ŷ k , ŷ k+1 , … are applied as inputs
xk+1 , xk+2 , …, respectively).
13.3 Python Script for Machine Translation by an RNN-Att
Figure 13.3 A many-to-many RNN-Att with multiple hidden and embedding layers suitable for machine
translation implementations.
vector sizes. Figure 13.3 shows connections of elements of a many-to-many RNN-Att with multiple
hidden and embedding layers suitable for machine translation implementations.
13.3 Python Script for Machine Translation by an RNN-Att
We now present a sample Python implementation for a many-to-many RNN with attention implementing a machine translator from German to English.
In line 23, the number of samples to be used to train the RNN-Att is defined as 3000. In line
24, batch size is defined as 16, which is the number of input samples to be handled at one
time at each epoch. In line 25, the number of hidden layers is defined as 512, which indicates
that the RNN-Att employs 512 encoders operating in parallel. For German, the sample size is 3000
sentences containing |G | = 1576 unique German words. However, in their translated 3000 English
sentences, there are |E | = 896 unique words. In line 26, the number of epochs is defined as 1000 for
training the RNN-Att. In line 27, the dimension of the embedding layer is defined as 256, implying
that the dimensions of E will be 256 × 1576. As in the case of a regular RNN using embedding, the
dimensions for a single input sample are reduce to 256 × 1 after embedding is applied.
In lines 29–42, the function named tokenizer accepts the sample data as input, converts the
words into numeric tokens and adds padding based on a maximum length (selected as 10 here) to
make them all appear equal in length. In line 33, it creates an alphabet called lang_tokenizer
based on the input samples. In line 37, the alphabet is encoded using sparse one-hot encoding [92]. In lines 39–41, padding is added to the encoded alphabet. In lines 44–50, the function
named load_dataset accepts the sample inputs and expected outputs and processes them by
calling the tokenize function. Outputs of load_dataset are data_input as the encoded
211
212
13 RNNs with Attention for Machine Translation
and padded sample inputs (i.e. the X set), data_target as the expected outputs (i.e. the Y set),
input_tokenizer as the input alphabet (in German) and target_tokenizer as the expected
output alphabet (in English).
In lines 52–70, create_rnnwAtt_model constructs an object for an RNN with attention. In
line 56, an input layer is defined. In line 58, an embedding layer is created using the alphabet size
(i.e. 1576) and embedding dimension. In line 60, an encoder with hidden layers is created. In lines
63–64, an attention layer is defined using all current and previous hidden states. In lines 66 and
67, a decoder is created using the expected output alphabet size for English, and an output layer is
defined. In line 69, an RNN-Att object called model is built using the input, embedding, encoder,
attention and decoder objects.
In lines 72–99, the predict function is defined. For a given input sentence, start and end tokens
are placed in line 75. In lines 77 and 80, respectively, input words are converted into numeric vectors
and padding is added. In line 83, the input sentence is converted to a tensor flow, which is a data
type to be used in the RNN-Att model. In line 85, the outputs of the softmax unit are assigned
to a vector called prediction. In line 90, the index of the highest probability value in prediction
is found by the tf.argmax function. In lines 95 and 96, the tokens are converted to words. After
removing the tags, the predicted sentence is returned.
In lines 101–106, the input samples are read from a file and split as inputs and corresponding outputs. In lines 109 and 110, the inputs and outputs are loaded into matrices called
input_tensor_train and target_tensor_train, respectively, by the load_dataset
function. In lines 117 and 118, an RNN-Att model is created using the parameters defined above:
input alphabet size (for German), target alphabet size (for English), embedding dimension and
number of hidden layers. In line 121, a tensor flow object called optimizer, which uses gradient
descent for optimization (i.e. the so-called Adam option) is created. In lines 122 and 123, the loss
function is defined as sparse cross-entropy loss [93]. In line 125, the model object
is compiled to include the optimizer and loss objects. In lines 135 and 136, the object called
model is trained using the input and output matrices, the number of epochs, the batch size
and the choice for the shuffle operation (if true, the order of input samples is changed at each
epoch). The remaining lines between 127 and 199 are for a simplistic user interface to train a
new RNN-Att, use a previously saved one, handle user queries either directly on screen or from a
previously recorded file, and store the results and the trained RNN-Att.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# The City College of New York, City University of New York
# Written by: Hasan Suca Kayman
# June 2024
# RNNwAtt_Translate is a simple translator that uses an RNN
# with Attention mechanism to translate DEU to ENG. Training
# matrials is a set of translated sentences taken from the Internet.
# Importing necessary libraries
import tensorflow as tf
from datetime import datetime
import os
import time
# Setting a random seed for reproducibility
seed = 301
tf.keras.utils.set_random_seed(seed)
# Example queries file
queries_path = 'queries_words.txt'
# Data input file
data_path = 'cleaned_words.txt'
13.3 Python Script for Machine Translation by an RNN-Att
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
noof_samples = 3000
batch_size = 16
noof_hidden_layers = 512
noof_epochs = 1000
embedding_dim = 256
# Tokenize a list of sentences and pad them to ensure equal length
# Convert words into numberic tokens and add padding based on max length
def tokenize(lang):
# Create a tokenizer with no filters
lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
# Fit tokenizer to the given text
lang_tokenizer.fit_on_texts(lang)
# Convert text to sequences of numbers
tensor = lang_tokenizer.texts_to_sequences(lang)
# Pad sequences
tensor = \
tf.keras.preprocessing.sequence.pad_sequences(tensor,\
padding='post', maxlen=10)
return tensor, lang_tokenizer
# Load a dataset and tokenize both input and target (expected) languages
def load_dataset(inp_lang, targ_lang):
# Tokenize both the input and target languages
data_input, input_tokenizer = tokenize(inp_lang)
data_target, target_tokenizer = tokenize(targ_lang)
# Return tokenized inputs, targets, and their tokenizers
return data_input, input_tokenizer,data_target, target_tokenizer
# Creates a simple RNNwAtt model with specified parameters
def create_rnnwAtt_model(inp_alph_size, targ_alph_size, embedding_dim, \
noof_hidden_layers):
# Define a sequential model with embedding, RNNwAtt, and output layer
inputs = tf.keras.Input(shape=(None,))
# Embedding layer
embedding = tf.keras.layers.Embedding(inp_alph_size, embedding_dim)(inputs)
# Simple RNN layer
rnn_output = tf.keras.layers.SimpleRNN(noof_hidden_layers, \
return_sequences=True)(embedding)
# Attention layer
attention_output = tf.keras.layers.Attention(use_scale=False, \
score_mode='dot')([rnn_output, rnn_output])
# Dense output layer
output = tf.keras.layers.Dense(targ_alph_size, activation='softmax')\
(attention_output)
# Build and compile the model
model = tf.keras.Model(inputs=inputs, outputs=output)
return model
# Given an input sentence, generate a response using the model
def predict(sentence, model, inp_tokenizer, targ_tokenizer, max_length=20):
# Preprocess the input sentence
preprocessed_sentence = '<start> ' + sentence.lower() + ' <end>'
# Tokenize the sentence and pad it to the max_length
input_sequence = inp_tokenizer.texts_to_sequences([preprocessed_sentence])
input_tensor = \
tf.keras.preprocessing.sequence.pad_sequences(input_sequence, \
maxlen=max_length, padding='post')
# print(inp_tokenizer.sequences_to_texts(input_sequence))
# Convert to a tensor and reshape for the RNN model
input_tensor = tf.convert_to_tensor(input_tensor)
# Generate predictions
prediction = model.predict(input_tensor, verbose=0)
# Extract the most likely tokens and convert them back to words
predicted_sequence = ['<start> ']
for pred in prediction[0]:
# Get the token with the highest probability
predicted_id = tf.argmax(pred, axis=-1).numpy()
# Stop if we reach the end token
if targ_tokenizer.index_word.get(predicted_id, '') == " <end>":
break
# Convert token to word
predicted_word = targ_tokenizer.index_word.get(predicted_id, '')
213
214
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
13 RNNs with Attention for Machine Translation
predicted_sequence.append(predicted_word)
# Return the original input and the predicted output and removing tags
result =' '.join(predicted_sequence).split("<start>")[-1].split("<end>")[0]
return result, sentence
# Read train data from a text file
file = open(data_path,'r').read()
# Split the train data into DEU and ENG sentences
qna_list = [f.split('\t') for f in file.split('\n')][:-1]
german_sentences = [x[0] for x in qna_list][:noof_samples+1]
english_sentences = [x[1] for x in qna_list][:noof_samples+1]
# Load and tokenize the dataset
input_tensor_train, input_tokenizer, targ_tensor_train, targ_tokenizer, = \
load_dataset(german_sentences, english_sentences)
# Set the unit count for alphabet sizes
inp_alph_size = len(input_tokenizer.word_index)+1
targ_alph_size = len(targ_tokenizer.word_index)+1
print(targ_alph_size)
# Create the RNN model with specified parameters
model = create_rnnwAtt_model(inp_alph_size,targ_alph_size, embedding_dim, \
noof_hidden_layers)
# Define the optimizer and loss function for training
optimizer_a = tf.keras.optimizers.Adam()
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True, reduction='none')
model.compile(optimizer=optimizer_a, loss=loss_object)
# prompt user to train or load an RNN model
print('\nOPTIONS:')
print('1 - Train a new RNN model\n2 - Load an existing model')
option = input('\nSelect an option by entering a number: \n')
if option == '1':
# OPTION 1: TRAIN A NEW RNN MODEL
print('\n********* NOW TRAINING A NEW RNN-Att MODEL *********')
model.fit(input_tensor_train, targ_tensor_train, epochs=noof_epochs, \
batch_size = batch_size, shuffle=True)
print('\n\n********** RNN-Att training complete **********\n\n')
elif option == '2':
# OPTION 2: LOAD RNN MODEL FROM FILE
message = 'Enter the file name of the RNN-Att Model you want to load: \n'
load_file = input(message)
# load the RNN model from load_file
model.load_weights(load_file)
print('\n\n****** SUCCESSFULLY LOADED RNN-Att MODEL ', load_file,'******')
# prompt user to test or save an RNN model
option_list = ['1','2','3','4']
while option != '4':
option = ' '
print('\nOPTIONS:')
print('1 - Test this RNN-Att model with input')
print('2 - Test queries given in a file')
print('3 - Save existing model and logs')
print('4 - Exit')
option = input('\nSelect an option by entering a number: \n')
if option == '1':
# OPTION 1: TEST MODEL
msg = 'Leave space before punctuation\n'
msg += 'Make sure your words are in the dictionary\n'
msg += 'Enter your input: '
user_input = input(msg)
result, sentence = \
predict(user_input, model, input_tokenizer, targ_tokenizer)
13.3 Python Script for Machine Translation by an RNN-Att
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
print('German Input: {}'.format(sentence))
print('Predicted English Translation: {}'.format(result))
elif option == '2':
# OPTION 2: TEST QUERIES
# List of queries for the model to respond to
file = open(queries_path,'r').read()
queries = [f.split('\t') for f in file.split('\n')]
#print(queries)
german_sentences = [x[0] for x in queries]
english_sentences = [x[1] for x in queries]
for index,query in enumerate(german_sentences):
result, sentence = predict(query, model, input_tokenizer,\
targ_tokenizer)
print('{}. German Input: {}'.format(index+1,sentence))
print('{}. Correct English Translation: {}'.\
format(index+1,english_sentences[index]))
print('{}. Predicted English Translation: {}'.\
format(index+1,result))
elif option == '3':
# OPTION 3: SAVE LOGS AND MODEL
times = datetime.now().strftime("%Y_%m_%d_%H_%M_%S")
model_path = 'RNNwAtt_MODEL_N{}_B{}_U{}_EP{}_EM{}_{}.h5'.format(
noof_samples,
batch_size,
noof_hidden_layers,
noof_epochs,
embedding_dim,
times
)
model.save_weights(model_path)
print("Succesfully Saved {}".format(model_path))
Figure 13.4 shows a subset of the training sample data used for our RNN-Att implementing a simplistic German to English translator. Each sample is a pair of sentences, where the first sentence
is in German and the second is its translation into English. A set of 10 responses generated by our
Figure 13.4 A subset of training data for our RNN-Att implementing a simplistic German to English
translator, where each sample is a pair of sentences consisting of a sentence in German and its translation
to English.
215
216
13 RNNs with Attention for Machine Translation
Figure 13.5 Sample responses generated by the Python script presented in Section 13.3: all user-entered
German sentences are unique (i.e. the individual words exist in the sample data, but the sentences do not).
RNN-Att implemented in Python script given above is displayed in Figure 13.5. All user-entered
German sentences are unique (i.e. different than the sentences in the sample data) but are formed
by the words that are present in the sample data. We see that our simplistic translator has generated remarkably accurate responses for some of the inputs.
13.4 Matlab Script for Machine Translation by an RNN-Att
In this section we present a sample Matlab implementation of the machine translator example
described in Section 13.3. The variables, functions and overall structure of the Matlab script given
below are kept similar to the sample Python script in Section 13.3 for ease of comparison.
In lines 17–21, the number of samples, batch size, hidden layers, epochs and embedding dimension are defined. In lines 30–44, the data is split into German and English sentences, which are
saved in variables called german_sentences_ and english_sentences_, respectively (both
are cell array types). In lines 46–49, a function named load_dataset is called with German
and English sentences, which returns encoded and padded cell arrays for inputs, expected outputs
(i.e. targets), a unique word list for outputs, and dictionaries. In lines 51–65, for the target cell array,
first the alphabet size (there are 896 English words for this example) is found and then the target
cell array is converted into a categorical cell array, which is a special data type required
by Matlab libraries.
In lines 67 and 68, the create_rnn_model_with_attention function is called with target
alphabet size, embedding layer dimension and number of unique words in English, which creates
an RNN with attention mechanism, using the embedding and hidden layers and alphabets.
13.4 Matlab Script for Machine Translation by an RNN-Att
In lines 70–74, training parameters are set for a gradient descent algorithm (i.e. the so-called
Adam option), the number of epochs, the batch size, shuffle every epoch mode (i.e. the order
of samples is changed at every epoch) and type of hardware execution environment (i.e. GPU, CPU,
etc.). In lines 76–163, a simplistic user interface is implemented for running the script. In lines 85
and 96, for the user-selected option of 1, a new RNN-Att model is trained using the input and
target tensors, layers and options as defined above. For option 2, a previously trained model is
loaded at line 91. After loading an RNN-Att model, the user is asked to either apply individual
queries from the command window in lines 108 to 112 (by entering 1 as the option); to run a set
of queries previously saved in the questions.txt file in lines 113 to 149; to save the current
queries, responses and model in lines 150 to 158 (by entering 3 as an option); or to exit the program
in lines 159 to 161 (by entering 4).
In lines 165–184, a function named load_dataset is implemented, whose parameters
include the input and target dictionaries. It generates encoded input data based on the German
dictionary and the target data based on the English dictionary, called input_tensor and
target_tensor, respectively. It also generates a word-to-numeric mapping for the input,
called lang_tokenizer, and a numeric-to-word mapping for the target, called reverseTokenizer. In lines 186–193, a function named tokenDictionary performs padding and
creates dictionaries and unique word lists for the input and target languages. In lines 195 to 212,
the create_rnn _model _with_attention function creates an RNN-Att model. It defines
the input, embedding and many-to-many encoder layers in lines 200, 201 and 202, respectively.
A function named DotProductAttentionLayer is called in line 204, which implements the
attention layer. In lines 205–207, a softmax-based decoder is defined, which finds the index of
the highest probability value. In lines 208–211, the decoder layer is defined using the number of
unique words in the target language (i.e. English) with all weights equal to 1, and the padding
coefficient is set to a very small number to prevent the padding character being used as part of
a prediction (otherwise, consecutive padding characters appear like a legitimate translation).
In lines 214 to 229, a function named predictResponse processes a user query, applies it to
a trained RNN-Att model and generates an output sentence using the mappings set up above.
In lines 231 to 244, the decoder function converts sequences that are sparse one-hot encoded
into words in the target language. In lines 246 to 257, a function named encoder applies sparse
one-hot encoding word by word to an input sentence.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
% The City College of New York, City University of New York
% Written by: Hasan Suca Kayman
% July 2024
% ml_translate_RNNwAtt is a simple machine translator that uses a
% Recurrent Neural Network with an Attention mechanism to translate from
% German to English. The training materials are a set of sentence pairs
% in German and English taken from the Internet.
% Setting a random seed for reproducibility
rng(301)
% Example queries file
queries_path = 'queries_words.txt';
% Data input file
data_path = 'cleaned_words.txt';
noof_samples = 3000;
batch_size = 16;
noof_hidden_layers = 256;
noof_epochs = 1000;
embedding_dim = 128;
217
218
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
13 RNNs with Attention for Machine Translation
% Read dialogs from a text file
fileID = fopen(data_path, 'r');
textData = textscan(fileID, '%s', 'Delimiter', '\n');
fclose(fileID);
% This gives us a cell array of strings (each line from the file)
textData = textData{1};
% Converting data type to cell
qna_list = cell(size(textData));
german_sentences_ = cell(noof_samples+1, 1);
english_sentences_ = cell(noof_samples+1, 1);
% Split each line into german sentences and english sentences
for i = 1:numel(textData)
qna_list{i} = strsplit(textData{i}, '\t');
end
% Extract german sentences and english sentences
for i = 1:noof_samples+1
german_sentences_{i} = qna_list{i}{1};
english_sentences_{i} = qna_list{i}{2};
end
% Load and tokenize the dataset
[input_tensor_train, target_tensor_train, ...
lang_tokenizer,rev_tokenizer,unique_words] = ...
load_dataset(german_sentences_, english_sentences_);
% Set the unit count for alphabet sizes
alph_size = 0;
for i = 1:noof_samples+1
if max(target_tensor_train{i}(:)) > alph_size
alph_size = max(target_tensor_train{i}(:));
end
end
% Add one for padding token
alph_size = alph_size + 1;
% Converting data type to categorical
for idx = 1:numel(target_tensor_train)
target_tensor_train{idx} = categorical(target_tensor_train{idx},...
0:alph_size-1);
end
% Create the RNNwAtt model with specified parameters
layers = create_rnn_model_with_attention(alph_size, ...
embedding_dim, noof_hidden_layers,unique_words);
% Training the model
options = trainingOptions('adam', 'MaxEpochs', noof_epochs, ...
'MiniBatchSize', batch_size, ...
'Shuffle', 'every-epoch', ...
'ExecutionEnvironment','cpu');
% User interaction for training or loading a model
disp('OPTIONS:');
disp('1 - Train a new RNN-Att model');
disp('2 - Load an existing model');
option = input('Select an option by entering a number: ', 's');
switch option
case '1'
disp('********* NOW TRAINING A NEW RNN-Att MODEL *********');
model = trainNetwork(input_tensor_train, target_tensor_train, ...
layers, options);
disp('********** RNN training complete **********');
case '2'
modelFileName = input(['Enter the file name of the' ...
' RNN-Att Model you want to load: '], 's');
load(modelFileName, 'model');
disp(['****** SUCCESSFULLY LOADED RNN-Att MODEL ', ...
modelFileName,'******']);
otherwise
error('Invalid input: Input must be 1 or 2');
13.4 Matlab Script for Machine Translation by an RNN-Att
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
end
% User interaction for testing or saving a model
while true
disp('OPTIONS:');
disp('1 - Test this RNN-Att model with input');
disp('2 - Test queries given in a file');
disp('3 - Save existing model and logs');
disp('4 - Exit');
option = input('Select an option by entering a number: ', 's');
switch option
case '1'
user_input = input('Enter your input: ', 's');
predicted_sentence = predictResponse(model, lang_tokenizer, ...
rev_tokenizer, string(user_input));
disp(['Predicted English Translation: ', predicted_sentence]);
case '2'
fileID = fopen(queries_path, 'r');
queries = textscan(fileID, '%s', 'Delimiter', '\n');
queries = queries{1};
fclose(fileID);
% Initialize cell arrays for storing sentences
german_sentences = cell(length(queries), 1);
english_sentences = cell(length(queries), 1);
% Split each line into German and English sentences
for i = 1:length(queries)
split_data = strsplit(queries{i}, '\t');
german_sentences{i} = split_data{1};
if numel(split_data) > 1
english_sentences{i} = split_data{2};
else
% Handling cases where there is no tab character
english_sentences{i} = '';
end
end
% Convert cell arrays to string arrays
german_sentences = string(german_sentences);
english_sentences = string(english_sentences);
for index = 1:length(german_sentences)
if isempty(german_sentences{index})
continue;
end
% Prediction function
predicted_sentence = predictResponse(model, ...
lang_tokenizer, rev_tokenizer,german_sentences(index));
% Display the results
fprintf('%d. German Input: %s\n', index, ...
german_sentences(index));
fprintf('%d. Correct English Translation: %s\n', ...
index, english_sentences(index));
fprintf('%d. Predicted English Translation: %s\n', ...
index, predicted_sentence);
end
case '3'
dtw = datestr(now, 'yyyy_mm_dd_HH_MM_SS');
model_path =sprintf...
('RNNwAtt_model_N%d_B%d_U%d_EP%d_EM%d_%s.mat', ...
noof_samples, batch_size, noof_hidden_layers, ...
noof_epochs, embedding_dim, dtw);
save(model_path, 'model');
disp(['Successfully saved to',model_path]);
case '4'
disp('Exiting program');
break;
end
end
function [input_tensor, target_tensor, lang_tokenizer,reverseTokenizer, ...
unique_words] = load_dataset(inp_lang, targ_lang)
% Create dictionaries for inputs and targets
[lang_tokenizer, ] = tokenDictionary(inp_lang);
219
220
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
13 RNNs with Attention for Machine Translation
[targ_lang_tokenizer, reverseTokenizer, unique_words] = ...
tokenDictionary(targ_lang);
% Initialize cell arrays for storing sentences
input_tensor = cell(length(inp_lang), 1);
target_tensor = cell(length(targ_lang), 1);
% Encode all of input and target data
for i = 1:length(inp_lang)
input_tensor{i} = encoder(lang_tokenizer, inp_lang{i}, 10);
target_tensor{i} = encoder(targ_lang_tokenizer, targ_lang{i}, 10);
end
% Transpose them to correct dimensions
input_tensor = input_tensor';
target_tensor = target_tensor';
end
function [tokenizer,reverseTokenizer,unique_words] = tokenDictionary(texts)
joined_texts = join(string(texts));
unique_words = unique(strsplit(joined_texts));
unique_words = ['<pad>', unique_words];
indices = 1:numel(unique_words);
tokenizer = containers.Map(unique_words, indices);
reverseTokenizer = containers.Map(indices, unique_words);
end
% Create a RNNwAtt model with specified parameters
function layers = create_rnn_model_with_attention(alph_size, ...
embedding_dim, rnn_units,unique_words)
ClassLayer = classificationLayer;
layers = [
sequenceInputLayer(1)
wordEmbeddingLayer(embedding_dim, alph_size)
lstmLayer(rnn_units, 'OutputMode', 'sequence')
% Dot product attention layer
DotProductAttentionLayer('attention')
fullyConnectedLayer(alph_size)
softmaxLayer
ClassLayer];
ClassLayer.Classes = unique_words;
ClassLayer.ClassWeights = ones(1, alph_size-1);
% Ignore padding while calculating loss
ClassLayer.ClassWeights(1.0) = 0.0001;
end
function predicted_sentence = predictResponse(model, lang_tokenizer, ...
rev_tokenizer, user_input)
% Preprocess the input sentence
preprocessed_sentence = '<start> '+ lower(user_input)+ ' <end>';
% Tokenize the sentence and pad it to the max_length
input_sequence = encoder(lang_tokenizer, preprocessed_sentence, ...
length(strsplit(preprocessed_sentence)));
% Generate predictions
predictions = classify(model, input_sequence);
% Convert tokens back to words
predicted_words = decoder(rev_tokenizer, predictions);
% Remove start and end tags and join into a single sentence
predicted_sentence = strrep(predicted_words,'<start> ','');
predicted_sentence = strrep(predicted_sentence,' <end>','');
predicted_sentence = strrep(predicted_sentence,'<pad>','');
end
function text = decoder(reverseTokenizer, sequence)
% Sequences that are in sparse one-hot encoding are converted to
% words:
words = cell(1, numel(sequence));
for i = 1:numel(sequence)
intVal = double(string(sequence(i)));
if intVal ∼= 0 && reverseTokenizer.isKey(intVal)
words{i} = reverseTokenizer(intVal);
else
words{i} = '';
end
13.4 Matlab Script for Machine Translation by an RNN-Att
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
end
text = strjoin(words);
end
function sequence = encoder(tokenizer, text, max_length)
% Word by word, sparse one-hot encoding is applied to sequences that
% include words:
words = strsplit(text);
sequence = zeros(1, max_length);
sequence(:) = tokenizer('<pad>');
for i = 1:length(words)
if isKey(tokenizer, words{i})
sequence(i) = tokenizer(words{i});
end
end
end
We now present the implementation of the DotProductAttentionLayer class for constructing the attention mechanism, which is a skeleton function defined in Matlab. This function was customized for our example here such that first the hidden state values are computed using Eq. (12.1),
then the attention weights as defined bt Eq. (12.2) (lines 36 to 40) and finally the context vector as
shown in Eq. (12.3) (lines 42 to 53).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
classdef DotProductAttentionLayer < nnet.layer.Layer
methods
function layer = DotProductAttentionLayer(name)
layer.Name = name;
layer.Description = "Dot product attention";
end
function context_vectors = predict(∼, hs)
[hiddenLayer, batchSize, num_time_steps] = size(hs);
% disp(['Shape of X: ', mat2str(size(X))]);
if (batchSize == 1 && num_time_steps == 1)
% For Matlab Test
context_vectors = hs;
else
attentions = cell(1, num_time_steps);
% Store context vectors for each time step
context_vectors = dlarray(zeros(hiddenLayer, batchSize,...
num_time_steps, 'single'));
for t = 1:num_time_steps
% Calculate attention scores
attention_scores = zeros(batchSize, t, 'single');
for tau = 1:t
% Compute dot product of hidden states for each
% batch
for b = 1:batchSize
attention_scores(b, tau) = dot(hs(:, b, t), ...
hs(:, b, tau));
end
end
% overflow check
max_scores = max(attention_scores, [], 2);
attention_scores = attention_scores - max_scores;
% Apply softmax to attention scores
attention_weights = exp(attention_scores);
attention_weights = attention_weights ./ ...
sum(attention_weights, 2);
attentions{t} = attention_weights;
% Compute context vector as weighted sum of hidden
% states
context_vector = zeros(hiddenLayer,batchSize,'single');
for tau = 1:t
221
222
46
47
48
49
50
51
52
53
54
55
56
57
58
13 RNNs with Attention for Machine Translation
% Multiply attention weights with corresponding
% hidden state
context_vector = context_vector + (hs(:, :, tau)...
.* attention_weights(:, tau)');
end
% Store the context vector for this time step
context_vectors(:, :, t) = context_vector;
end % for
end % if
end % func: predict
end % method: nnet.layer.Layer
end % class: DotProductAttentionLayer
For a set of user queries, the responses generated by our RNN-Att implemented in Matlab are
shown in Figure 13.6. None of the input German sentences is in the sample data set, although
they use the German words contained in the sample data. Considering that this is a very simplistic
model, using a sample set of few thousand sentences (as opposed to millions of them) and trained
Figure 13.6 Sample responses generated by Matlab script presented in Section 13.4: all user-entered
German sentences are unique (i.e. the individual words exist in the sample data, but the sentences do not).
13.6 Exercises for Chapter 13
by a small number of epochs, the English translations generated by the above Matlab script are
remarkably close to their correct translations. This result implies that RNN models supported by
attention mechanisms are powerful tools and that they can be feasibly implemented for real-life
applications.
13.5 Concluding Remarks
In this chapter we focus on machine translation applications using RNNs with the attention mechanism. Because even the last words in a sentence may change its translation to another language,
output generation does not start until all words in the input sentence have been processed by the
RNN-Att. Another feature of machine translation is that the length of an input sentence may be
different to its translation. Such characteristics necessitate a many-to-many RNN-Att architecture.
Because natural languages tend to have large vocabularies and the size of the sample data may be
in millions, RNN-Att architectures with multiple hidden and embedding layers are customary for
machine translators.
13.6 Exercises for Chapter 13
1) For the Python implementation presented in Section 13.3, if you increase the number of
epochs, how would training time and prediction power (i.e. compare the responses to the
same user queries) be impacted?
2) For the Python implementation presented in Section 13.3, if you increase the number of hidden
layers, how would training time and prediction power (i.e. compare the responses to the same
user queries) be impacted?
3) Repeat Exercise 1 for increasing batch size.
4) Repeat Exercise 1 for increasing embedding layer size.
5) If you train the RNN in the example presented in Section 11.4 with a dataset that has roughly
twice as many elements, what will be the impact on training cost and prediction power.
6) Repeat Exercise 1 for the Matlab implementation presented in Section 13.4.
7) Repeat Exercise 2 for the Matlab implementation presented in Section 13.4.
8) Repeat Exercise 3 for the Matlab implementation presented in Section 13.4.
9) Repeat Exercise 4 for the Matlab implementation presented in Section 13.4.
10) Repeat Exercise 5 for the Matlab implementation presented in Section 13.4.
223
225
14
Genetic Algorithms
14.1 Introduction
Optimization is different from finding an absolute maximum or minimum for a given constraint
in that it finds the most suitable solution for the problem under a set of constraints. For non-linear,
discrete or discontinuous space problems, which are the characteristics of many real-life applications in the twenty-first century, finding an optimal solution is often a challenging task. Most optimization problems associated with current engineering and managerial tasks tend to involve a
large number of requirements typically conflicting with each other. A passenger plane on its route
to Hawaii aims to optimize fuel consumption by selecting the shortest path, while avoiding thick
clouds on its way to reduce turbulence which may cause extra stress to its wings, but without sharp
movements that may make passengers uncomfortable. A manager in charge of manufacturing a
brain-monitoring chip to reduce the effects of Parkinson’s disease on elderly patients has to construct a team with doctors, engineers, scientists, chip manufacturers, legal experts, and sales and
marketing experts from a pool of candidates with different experience levels and salary requirements, while staying within a given budget. They will be forced to make decisions with adverse
effects on their project since they probably cannot afford to choose only the top candidate from
each field due to budgetary restrictions or their availability. Every movie producer, football team
manager, or space mission commander faces these types of challenges daily. By introducing extra
layers of complexity, modern tasks necessitate that a new problem-solving framework is employed
since these requirements are typically beyond the scope of conventional optimization methods.
Although satisfying multiple conflicting criteria is a formidable challenge for traditional methods,
multi-objective optimization problems are excellent candidates for biologically inspired
(or bio-inspired) computation techniques, which are designed to mimic processes found in
nature. Typical examples of biologically inspired methods include:
survival of the fittest: in a given population, only the better (fitter) individuals survive, usually the ones who can adapt to dynamically changing environmental conditions.
● swarm behaviour: a flock of birds acts as a single entity for tasks vital to the livelihood of the
flock; for example, if one of the birds in a flock sees a good food source while flying, all birds of
the flock benefit from this knowledge.
● evolution of species: in a given population of species, for survival, individuals evolve to
adapt to changing environmental conditions; for example, birds in rocky islands grow long beaks
to pick worms from between rocks.
●
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
226
14 Genetic Algorithms
●
social behaviour: survival among different species and within a population of the same
species favour those with even the slightest advantage, and hence there is fierce competition
between different species as well as among the members of a given species.
Being satisfied with an optimal or a near-optimal solution (as opposed to a global minimum or
maximum), bio-inspired algorithms are capable of operating in otherwise prohibitively large problem spaces by emulating an evolutionary process observed in nature. Characteristics of bio-inspired
techniques include:
handling non-differentiable, nonlinear and multi-modal problems;
parallelization to cope with computation-intensive cost functions;
● observability of control variables (e.g. convergence, fitness, diversity); and
● low computational complexity provided that they are setup properly.
●
●
The most popular bio-inspired computation methods include:
genetic algorithms [104] (Chapters 14, 15, 16 and 17)
● particle swarm optimization [105] (Section 18.2 in Chapter 18)
● differential evolution [106] (Section 18.3 in Chapter 18) and
● grammatical evolution [107] (Section 18.4 in Chapter 18).
●
In the rest of this chapter, we introduce the basic elements of genetic algorithms together with
their basic operation and functionality.
14.2 Genetic Algorithm Elements
Genetic algorithms (GAs) mimic the evolution processes observed in nature. Hence, biological
terms are embraced to describe these parallels. A so-called chromosome represents a possible
solution to a given problem and is referred to as an individual. Typically, a chromosome consists of indivisible building blocks called codons, whose values are updated as the GA progresses.
Below are a few examples for individuals in various GA applications:
a location to move for a mobile node
a price to buy or sell a specific financial stock
● a voltage to apply to a given port
● an angle to cut a specific machine part
● an amount of drug to add into a given medicine
● a period to wait until repeating an experiment
●
●
A collection of individuals at a given generation is called a population. Each individual in a
population independently seeks to find an optimal solution. The number of individuals in a population represents the amount of parallelism employed by the GA in a search space. Although
individuals of a population change at each generation (fitter children replacing less fit parents),
population size is usually kept constant throughout generations.
A selection mechanism picks parent chromosomes for a crossover operation that yields one
or more offspring. With a small probability, codons may be mutated (some of its properties are
randomly altered). A population evolves from one generation to the next by selection, crossover
and mutation operations. A mathematical function, called fitness, provides a numerical representation of the proximity of a candidate solution to the overall optimization goal. Determining an
appropriate fitness function for a given problem is crucial in all bio-inspired algorithms.
14.3 A Simple Algorithm for a GA
Figure 14.1
Examples of fitness functions and individuals with good and bad fitness.
Examples of various fitness functions and individuals with good and bad fitness are presented
in Figure 14.1. In the first one, suppose a UAV (unmanned aerial vehicle or a drone) operating in
a swarm of UAVs is using a flight control algorithm based on a GA in deciding which location to
move in the next time step. In this case, a chromosome represents the possible locations to which
the UAV can move. For a candidate position to have a good fitness value, it must take the UAV away
from a crowded region of UAVs but not disconnect it from its neighbours. A position with a bad
fitness would leave the UAV in a region congested with other UAVs or in an isolated location with
no neighbours. In the next example, a chromosome represents a decision to buy or sell a financial
stock. A candidate solution that sells some of the stocks and buys other stocks and, at the end,
makes a profit has good fitness. A solution that loses money after a transaction has bad fitness. The
amount of profit represents the goodness of a solution among several good ones (i.e. the candidate
that makes the most profit is the best one). Similarly, in the last example, a GA is used in deciding
which anti-cancer drug amount is to be given to a patient. If a candidate solution causes the tumour
to shrink, it has a good fitness. The amount of shrinkage may be part of the criteria to decide which
solutions are better than others.
14.3 A Simple Algorithm for a GA
Let us now present the basic operations of a GA as shown in Figure 14.2:
Evaluation: Goodness of each individual is assessed by a fitness function, which assigns a
numeric value for quantitative comparison of individuals.
● Selection: Based on a predetermined criteria, parents are selected to generate offspring (typically giving preference to better-performing individuals).
● Reproduction: Using the selected parents, offspring are generated and their fitness is
evaluated.
● Replacement: Children fitter than existing members of a population replace those lessperforming individuals as a new generation is formed.
●
In Figure 14.3 we present pseudo-code for implementing a simple GA [108]. First, an initial population of candidate solutions is formed, where chromosome values are set randomly. Then then
the fitness of each individual is computed. Based on fitness, parents are selected and offspring are
227
228
14 Genetic Algorithms
Figure 14.2
Basic operations of a GA.
Figure 14.3
A simple GA implementation.
formed by crossover and mutation operations. Finally, offspring are evaluated for their possible
inclusion in the next generation.
This process is repeated for a predetermined number of iterations or until a satisfactory solution is found. Every time better-performing offspring are included in a population, it is said that a
new generation is formed. There are several mechanisms to select offspring as part of a new
generation. The most popular selection mechanisms include roulette wheel selection,
where the fitness level of a chromosome is indicative of its chance of being selected as a parent for
the next generation, and elitist selection, in which the fittest individuals are advanced to
the next generation without modification, while typically keeping the ratio of the elite group to the
population size small.
The Single-point crossover operation is popular in generating offspring out of two
parent chromosomes, where one pivotal point bit is randomly selected in both parents, and
the data between parent chromosomes are swapped around this bit. To perform a single-point
crossover between two parents, each with a chromosome length of K bits, first a random integer
CR ∈ (0, K − 1) is selected. The first child is created by taking the most significant CR bits from
14.3 A Simple Algorithm for a GA
Figure 14.4
An example of a single-point crossover operation in a GA.
parent 1 and the least significant (K − CR) bits from parent 2. A second child is created by taking
the most significant CR bits from parent 2 and the least significant (K − CR) bits from parent 1.
In the example shown in Figure 14.4, a single-point crossover is performed for CR = 2 for two
parents, each with six-bit chromosomes, where child 1 has the top two bits from parent 2 and
the lower four bits from parent 1, whereas child 2 receives its top two bits from parent 1 and the
remaining ones from parent 2.
In different GA realizations, crossover operation can be implemented using multiple points or
with more than two parents. For example, in a two-point crossover with two parents, two points, CR1
and CR2 , are randomly selected to swap the parent chromosomes. Figure 14.5 shows an example
for a two-point crossover with CR1 = 2 and CR2 = 4. Compared to single-point crossover, two-point
crossover is preferable to keep a population with a diverse set of individuals for more effective
exploration of a search space.
Mutation is defined as a genetic operation that randomly changes one or more bits in a chromosome. With a small probability or using a predefined probability distribution model (e.g. Gaussian),
mutation introduces diversity into a population so that candidates do not get stuck at local extreme
points but instead fully explore a search space for global extreme points.
Figure 14.5
An example of a two-point crossover operation in a GA.
229
230
14 Genetic Algorithms
14.4 An Example of a GA
Let us now present a simplistic example illustrating the basic steps of a GA. In Figure 14.6, we
see a population of 8 individuals, whose chromosomes consist of 12 codons. In step 1, an initial
population of individuals with randomly selected codon values is formed. In step 2, the fitness of
each individual is computed, which is symbolically represented as a dollar sign. In this illustration, we assumed that the more dollar signs an individual has, the fitter it is. In step 3, the two
fittest individuals are selected as parents. In typical applications, multiple pairs of parents may be
selected to generate multiple children at each generation. For example, 20% of individuals may
be selected as parents, giving a preference to the fitter ones as described in Section 14.3. These
three steps correspond to the evaluation and selection operations in Figure 14.2.
Now it is time to generate offspring out of the selected parents as shown in Figure 14.7. With a
single-point crossover operation for CR = 4, two children are generated. Mutation is performed
for one child (whether it will be applied is determined by a mutation probability, which is typically
Figure 14.6
Initial population and selection of parents in a GA.
Figure 14.7
Reproduction of offspring in a GA.
14.5 Convergence in GAs
Figure 14.8
Selection in a GA.
a small value), whose codon 7 (determined randomly) is replaced with a random value (codon 7
was 51 and was changed to 22). These actions correspond to the reproduction operation in
Figure 14.2.
The fitness of newly generated children must be calculated and a decision must be made as
to whether they will be included in the next generation, as shown in Figure 14.8. In this simple
example, we assumed that children 1 and 2 have fitness values of 2 and 5 dollar signs, respectively.
As shown in Step 6 in Figure 14.8, both children are better than some of the existing individuals in
the population. Therefore, these children are included in the new generation, while two existing
non-performing individuals are dropped from the population (each with a fitness of 1 dollar sign).
These actions are represented as the selection operation in Figure 14.2.
At this point, our simple example has a new generation of individuals (with two new members).
The steps shown in Figures 14.6 to 14.8 are repeated for the following generations until either an
acceptable result is obtained (e.g. an individual with 10 dollar signs is generated) or a predetermined
upper limit for the number of generations is reached (e.g. 20 generations).
14.5 Convergence in GAs
Convergence is a metric indicating if the solutions found by a GA from one generation to the next
are approaching an acceptable and good-quality solution. A desirable outcome is that the best
(or average) fitness of a population steadily improves. Sometimes, a bio-inspired method cannot
find a solution due to improper setting up of a problem. Possible reasons for a non-convergent GA
include:
the chromosome size is to small
● the number of generations is insufficient
● the population size is too small
● the fitness function cannot properly identify fit and unfit individuals
● the GA mechanism is not suitable to find a solution for this problem type
●
231
232
14 Genetic Algorithms
If one or more of these parameters are not set up properly, the GA may not converge to an optimal
(or a near-optimal) solution. Many analytical and numerical metrics can be defined to predict and
measure GA convergence, including:
the best fitness in the population improves
variance among the fitness of individuals diminishes
● fitness value oscillation through generations (if any) decreases or diminishes
●
●
As generations progress, we may observe signs of non-convergence:
population diversity decreases (i.e. all individuals look like each other)
fitness values start fluctuating from one generation to another
● solutions with better fitness are not distinguishable from the less fit ones
●
●
14.6 Concluding Remarks
Bio-inspired computation techniques are typically applied to problems that are intractable or computationally too intensive to be solved using classical methods (e.g. non-linear optimization problems). As one of the most popular bio-inspired techniques, GAs, with their relatively simple design,
low cost computational requirements and independence from software development platforms, are
perfect candidates for many real-life optimization problems in a wide variety of settings. For a successful implementation, one must be careful in selecting GA parameters, including chromosome
structure, parent selection mechanism, crossover method, mutation probabilities and fitness function definition.
We cover applications of GAs in the following chapters of this book: finding dietary menu selections under nutritional restrictions (i.e. solving the knapsack class of problems) in Chapter 15,
flight control of autonomous drones in Chapter 16, finding optimum routes for travel (i.e. solving the travelling salesman class of problems) in Chapter 17 and generating syntactically correct
programs for a given BNF grammar in Chapter 18.
14.7 Exercises for Chapter 14
1) Give an example of an unsolved problem in your field of expertise that you think could be solved
by GAs.
2) Suggest a GA structure for finding a suitable position for a UAV (unmanned aerial vehicle or a
drone) located at (0,0,10) coordinates with three neighbouring UAVs at (10,0,10), (10,10,10) and
(0,10,10) positions. In this notation, we assume that (x, y, z) corresponds to latitude, longitude
and altitude of a UAV position. Suppose this UAV can change its latitude and/or longitude
and/or altitude by 1 unit at a time. For example, some of the possible positions for this UAV
include (1,0,10), (1,1,11), (-1,1,9), and so on.
The goal is to find a flight control algorithm based on a GA that will move the UAV away from its
neighbours. You can assume for simplicity that while the UAV is making a decision, its neighbours do not move.
a) Suggest a chromosome.
b) Define a fitness function. (Hint: You may use distance to neighbours as part of the fitness
function such that candidate positions moving the UAV farther away from neighbours are
fitter than ones moving closer.)
14.7 Exercises for Chapter 14
c) How many individuals are needed in the population?
d) How many generations are needed for the GA?
e) Will your GA implementation generate an acceptable solution? How do you know?
3) Suggest a GA structure for finding the key for a 10-digit lock combination. Suppose that the
digits are binary (i.e. either 0 or 1) and the secret combination is given as 1 0 1 0 1 0 1 0 1 0.
You may assume that when a combination is applied to the lock, a visual indicator displays the
total number of matching digits (but not the positions of matching digits); if all 10 digits match,
the lock opens. For example, if one applies 1 1 0 0 1 1 0 0 1 1 to the lock, the indicator displays
5 since there are 5 matches as underlined: 1 1 0 0 1 1 0 0 1 1.
a) Suggest a chromosome.
b) Define a fitness function.
c) How many individuals are needed in the population?
d) How many generations are needed for the GA?
e) Will your GA implementation generate an acceptable solution?
f) Is this problem suitable for employing a GA to find the solution?
4) Suggest a GA structure for finding the key for an 8-digit lock combination. Suppose that the
digits are integers between 0 and 9, and the secret combination is given as: 1 9 9 6 2 0 0 1. You
may assume a visual indicator displaying the number of matches for a given candidate solution
is available and that the fitness function has direct access to it for any candidate solution.
a) Suggest a chromosome.
b) Suggest a fitness function.
c) How many individuals are needed in population?
d) How many generations are needed for the GA?
e) Will your proposed GA implementation converge to an acceptable solution?
f) Is this problem suitable for employing a GA to find the solution?
5) How do bees know to build hexagon-shaped honeycombs, not square- or triangle-shaped ones?
Did they take an optimization class in a civil engineering department for constructing optimal
structures?
6) Why do you think an octopus has eight legs, not nine or seven?
233
235
15
GAs for Dietary Menu Selection
15.1 Introduction
AI-based computational methods are capable of obtaining satisfactory solutions for otherwise
intractable problems if a proper model can be constructed for the problem. In this chapter,
we study a well-known computationally expensive problem, namely the so-called knapsack
problem (KP)1 [109], and present a computationally feasible solution using genetic algorithms.
KP is designed specifically to solve the types of challenges where decisions are choices that must
obey a set of limitations with the goal of maximizing the outcome of a task.
In the engineering domain, the KP represents a class of optimization problems where there are
multiple and often conflicting requirements to be selected to satisfy objectives such as to maximize
a profit or minimize a loss. Airlines and naval cargo ships have to select freight to maximize their
profits while staying within the weight and capacity limitations of a given flight. A portfolio manager considers potential investment opportunities and their expected returns to maximize gains
without exceeding their allocated budget. A project manager may need to select a team from a pool
of engineers with different levels of expertise and salaries, while including individuals with the
highest talent level and keeping the total cost as low as possible.
As an example, we choose a dietary menu selection problem, where a set of food items, each with
a calorie and protein value, will be selected such that the total calorie amount of the selected items is
below a given limit while the total protein amount is kept as large as possible. Consider the fictitious
cinema menu shown in Figure 15.1 [110]. Suppose one has to select items from this menu such that
total calorie count should stay below 1000 calories. This restriction implies that one cannot select
both medium plain popcorn (600 calories) and butter topping (480 calories) since the total calories
will be above the 1000 calorie limit. A valid solution, on the other hand, may include medium
plain popcorn with nacho cheese (160 calories), which has a total of 640 calories and 17 grams
of protein. However, selecting small plain popcorn (300 calories) and a hot dog (530 calories) is a
better solution since it has a total of 26 grams of protein. We can easily see that if the calorie limit
is higher or the number of food items is large, making a good menu selection will be a challenging
task even for such simple examples.
We can see from this simple example that to find a solution for a proper selection of menu items,
we have to try many combinations of items and check if they satisfy the dietary constraints. If there
are more items in the menu (e.g. in a typical dietary menu, there are hundreds of items), this selection becomes a formidable problem. In fact, the computational cost of the KP grows exponentially
1 Backpack in the UK
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
236
15 GAs for Dietary Menu Selection
Figure 15.1 Example menu items for a fictitious movie theatre menu [110]; Anthony Poynton / Public
Domain Pictures.
as the number of items grows linearly. Therefore, we cannot employ traditional methods to find
good solutions for the knapsack class of problems.
Application areas of the KP often go beyond engineering problems, spanning into business,
finance, manufacturing, task selection, project management and strategic decision making. For
example, when planning an investment, allocation of financial resources among stocks can
be mapped into a KP of maximizing long-term value. By representing expected returns, share
prices and budget as values, weights and the capacity of a knapsack, respectively, the priority
and importance of each share in a proposed investment strategy can be determined by solving a
KP [111]. The following list presents a small subset of typical applications of the KP:
Resource allocation problems, where a selection of non-divisible projects or tasks must be scheduled while staying under a given budget and certain time constraints.
● In metal processing facilities, finding the least wasteful way to cut raw materials
● Selection of nutritional items while maintaining various dietary restrictions with respect to calories, sugar and carbohydrates while maximizing proteins or fibre.
● Project team selection to maximize productivity while staying under a specific budget.
● Identifying mobile nodes for routing network connections to minimize delays and maximize
overall throughput.
● Power allocation management and optimized scheduling for construction sites, factories, oil
refineries, factory floors, etc.
●
15.2 Definition of the KP
Figure 15.2 shows a knapsack with a weight capacity of 15 kg, where five items are to be selected and
placed in it (example taken from [112]). Each item has a different weight and value (e.g. the item
15.2 Definition of the KP
Figure 15.2 A simple example of the KP – given a knapsack of
15 kg capacity, the goal is to place items into the knapsack such
that the total value is high and the total weight is less than
15 kg (Adapted from [112]).
Figure 15.3 Possible item selections to be
placed into the knapsack shown in Figure 15.2
with the assumption that only one copy of each
item is available.
with weight 12 kg has a value of 7 dollars, whereas the value for the item weighing 9 kg is 10 dollars).
The goal is to place a set of these items into the knapsack such that the total weight is less than 15 kg
while keeping the total value as high as possible. Figure 15.3 shows several item selections that can
fit into the knapsack. For example, the choice shown in Figure 15.2a has a total weight of 14 kg
with a cumulative value of 9 dollars. Among all the choices shown in Figure 15.3, we can see that
the most valuable selection is the one in Figure 15.3c.
This formulation is known as a knapsack problem (KP) [113]. The name knapsack problem was
coined by mathematician Tobias Dantzig in the early twentieth century [109]. As discussed further
in this chapter, there are different versions of the KP, where the items may have single or multiple copies with single or multiple constraints (e.g. weight and volume). For the example given in
Figure 15.4, if multiple copies of items are available, the solution is selecting three yellow and three
grey boxes with a total weight of 15 kg and a value of 36 dollars. If only a single copy of each item
can be selected, the solution includes all items except for the green box, with a total weight of 8 kg
and a value of 15 dollars.
For a given set of items, each with a weight and value, to be added into a knapsack which has a
limited capacity, the KP is defined as selecting items from this set such that the total value of the
selected items is maximized without exceeding its capacity. Since the solution space grows exponentially with the number of elements to be placed into the knapsack, the KP has been recognized
as a computationally difficult problem.
Let us consider a knapsack with a weight capacity of KW . Let there be n items to be loaded into
the knapsack, each item j with weight wj and a value of vj dollars. A good solution to the KP should
237
238
15 GAs for Dietary Menu Selection
Figure 15.4
Solutions for KP shown in Figure 15.2.
select a combination of items that can be carried in the knapsack while offering the highest possible
total value for the knapsack. Hence, a solution to the KP attempts to accomplish the following goal:
maximize
n
∑
j=1
xj ⋅ vj subject to
n
∑
wj ⋅ xj ≤ KW ,
(15.1)
j=1
where the binary variable xj is used to determine if an item j should be placed in knapsack or
not; hence xj ∈ {0, 1}, j = 1, 2, · · · , n. Since each item is either selected or not, this version of KP is
referred to as 0-1 KP.
15.3 A Simple Algorithm for the KP
If one uses a straightforward approach for solving the KP by testing all possible solutions to it, for
a KP with n items to select, there are a total of 2n combinations of items to be considered. A simple
algorithm with a similar approach for implementing 0-1 KP is shown in Figure 15.5, where all
possible assignments of items are considered, then the knapsack is packed with items that make it
the most valuable as the final solution.
As seen in Figure 15.5, the number of steps needed to find a solution to the KP is exponential
with respect to the number of items to be placed in the knapsack, as opposed to algorithms with
linear or polynomial growth with the input size. Types of problems that do not have linear- or
polynomial-time solutions are called NP-complete [114]. The amount of time to obtain a solution
to an NP-complete problem grows exponentially as the input size grows linearly, which makes such
problems prohibitively expensive for practical applications. The algorithm in Figure 15.5 has a complexity of O(n2n ) (i.e. for n items, it needs in the order of n ⋅ 2n steps to complete), implying that
addition of a single item to the set of items will double the time needed to complete its computation.
Therefore, the algorithm in Figure 15.5 would be suitable only to solve KP applications with very
small numbers of items. Considering that a realistic menu contains hundreds of food choices, computing KP solutions may easily require examining billions of combinations for even a modest-sized
real-life problem. Therefore, techniques based on bio-inspired algorithms are a necessity for solving
realistic knapsack class of problems in feasible time. In Section 15.5, we introduce a GA to solve the
KP in linear time, even though the solution may not be an absolute global minimum or maximum
but an approximation.
15.4 Variations of the KP
Figure 15.5
A simple algorithm to find the best combination of items for KP.
15.4 Variations of the KP
There are several variants of the KP, each tailored to solve a specific application. In one version,
multiple copies of an item are available for selection. In a situation where it is permissible to select
multiple copies of each item, Figure 15.6 shows possible items that can be placed into the knapsack
Figure 15.6 Possible selections of items to be placed into the knapsack shown in Figure 15.2 for KP
allowing repeated selections of the same item.
239
240
15 GAs for Dietary Menu Selection
of Figure 15.2. For example, the choice shown in Figure 15.6a has four copies of the 2-kg item. This
class of KP is called the KP with repetitions (KPR). Consider a disaster relief organization
with virtually unlimited resources but with limited transportation capabilities. This organization
will benefit from KPR solutions to select items to dispatch to a crisis area [115]. Let us define sj as
the maximum number of times that an item j can be selected for a knapsack with a capacity of Kw .
The KPR is then formally stated as
maximize
n
∑
xj ⋅ vj subject to
j=1
n
∑
wj ⋅ xj ≤ KW , 0 ≤ xj ≤ sj for j = 1, · · · n,
j=1
where xj is the number of times item j is selected (0 ≤ xj ≤ sj for j = 1, · · · n).
The KP that aims to satisfy multiple constraints at the same time, for example when both
limited weight and volume capacities of a knapsack have to be considered, is referred to as the
multidimensional KP (MKP). The objective of the MKP is to maximize the value of all items
that fit into a knapsack, while examining the weight and volume of each item before determining
its suitability for packing. There are many applications of the MKP in real-life tasks. A transportation company that must determine the selection of goods to be dispatched while considering the
volume and weight limitations of a truck may construct its problem as a MKP [116]. A similar
MKP application is dietary menu selection, where there are food items with different nutritional
characteristics (e.g. vitamins, fibres, calories), each with a lower or upper limit for a healthy
intake. If each food item is associated with a price, the objective would be to keep the cost within
a budgetary limit while adhering to multiple nutritional restrictions. Furthermore, there could be
a requirement for diversity in a diet such that a menu consisting of only vegetables may not be
desirable and a variety of items must be included in a successful selection. Formally, we can define
the MKP as
n
n
∑
∑
maximize
xj ⋅ vj subject to
wij ⋅ xj ≤ ci , for i = 1, · · · m,
j=1
j=1
where a set of n items are considered for a knapsack with m restrictions. For example, m = 2 could
reflect constraints c1 and c2 referring to weight and volume restrictions of a knapsack, respectively.
An item j with a value of vj ≥ 0 uses an amount wij ≥ 0 that is bound by constraint ci . In our
example of the MKP that has limited volume and weight capacities, w1j indicates the weight of
item j, and w2j represents its size.
15.5 GAs for KP Solution
Let us now illustrate how GAs can be employed to solve the knapsack class of problems requiring
computation cost that is linearly proportional to the number items. Suppose a list of food items
for this example is given in Figure 15.7, where each item has its calorie count and the amount of
protein it provides.
For a GA designed as a solution for KPs, a binary chromosome can be defined to represent the
selected items, where a chromosome bit is a 1 if the corresponding item will be included in the
knapsack and 0 otherwise. The fitness function can be defined as the total value of the selected
items only if the total weight is below the knapsack capacity. Only the chromosomes representing
valid selections of items are to be considered in further generations of the GA. For a set of n food
items, the chromosome will then be
[
]
x1 x2 … xn for xi ∈ {0, 1},
15.5 GAs for KP Solution
Figure 15.7
values.
Example list of food items and their calorie and protein
where xi = 1 if item i is selected and xi = 0 otherwise. Let us repeat the fitness function defined in
Eqn (15.1) for convenience:
maximize
n
∑
xi ⋅ pi subject to
i=1
n
∑
xi ⋅ ci ≤ KW .
i=1
The computation of the first summation in Eqn. (15.1) implies that the chromosome (which can be
considered as a one-dimensional matrix) and the column of the food table that contains the protein
values must be multiplied as follows:
[
protein = x1 x2
⎡ p1 ⎤
] ⎢ p2 ⎥
· · · xn ⋅ ⎢ ⎥ = x1 p1 + x2 p2 + · · · + xn pn .
⎢· · ·⎥
⎢p ⎥
⎣ n⎦
(15.2)
To find the total sugar in a given chromosome in this context means that the chromosome must be
multiplied by the sugar column of the food table:
[
calorie = x1 x2
⎡ c1 ⎤
] ⎢ c2 ⎥
· · · xn ⋅ ⎢ ⎥ = x1 c1 + x2 c2 + · · · + xn cn ≤ KW ,
⎢· · ·⎥
⎢c ⎥
⎣ n⎦
where ci is the calorie amount for a food item i taken from the food table given in Figure 15.7. As
an example, consider an individual defined as
[
]
1 0 1 0 1 0 0 0 0 1 .
The fitness for this individual is the matrix multiplication of the chromosome and the protein column of the food table:
⎡5⎤
⎢36⎥
⎢ ⎥
⎢21⎥
⎢21⎥
[
] ⎢36⎥
fitness = 1 0 1 0 1 0 0 0 0 1 ⋅ ⎢ ⎥ = 5 + 21 + 36 + 20 = 82.
⎢4⎥
⎢8⎥
⎢ ⎥
⎢1⎥
⎢1⎥
⎢ ⎥
⎣20⎦
241
242
15 GAs for Dietary Menu Selection
The total calorie value of this chromosome can be calculated using Eq. (15.1):
⎡ 17 ⎤
⎢168⎥
⎢ ⎥
⎢123⎥
⎢102⎥
[
] ⎢255⎥
fitness = 1 0 1 0 1 0 0 0 0 1 ⋅ ⎢ ⎥ = 17 + 123 + 255 + 79 = 474.
⎢ 80 ⎥
⎢ 80 ⎥
⎢ ⎥
⎢ 65 ⎥
⎢ 13 ⎥
⎢ ⎥
⎣ 79 ⎦
15.6 Python Script for Dietary Menu Selection Using a GA
In this section, we present a simple Python script implementing the dietary menu selection problem. In this script, a class named GeneticOptimizerKP is defined between lines 20 and 155.
In its constructor in lines 22–40, the parameters used in this class are initialized. Between lines
42 and 82, in the init_toolbox method, the GA parameters are defined, including individual,
chromosome values, fitness evaluation, crossover, mutation and selection operations.
The fitness function is presented in the method named fit_func in lines 88 and 89 as the
multiplication of a chromosome with the protein column of the input:
np.dot(chromosome, self.items_df[‘Protein’])
In the method named validate, in lines 95 and 96, chromosome validation is presented:
np.dot(self.A, chromosome) <= selb.b)
where A and b are defined in the method named fit presented between lines 107 and 160. In lines
116 and 117, the matrix A is defined as the Calories column of the input file, whereas b is set to
the calorie_capacity value. In line 20, the population is defined as
pop = self.toolbox.population(n=20)
where object pop contains 20 individuals. Selection of the best individual is defined in line 126:
self.best_chromosome = tools.HallOfFame(1)
which indicates that the individual with the best fitness is saved in the variable named
best_chromosome. In line 137, the GA parameters are defined as
algorithm.eaSimple(pop, self.toolbox, cxpb=0.7, mutpb=0.2,
ngen=50,... halloffame = self.best_chromosome, verbose = False)
where cxpb and mutpb are the crossover and mutation probabilities, respectively, and ngen is
the number of generations for which to run the GA. In lines 158 and 159, the calorie capacity and
the input file are set, respectively. In line 160, an object named optimizer is instantiated:
optimizer = GeneticOptimizerKP(calorie_capacity, items_file)
which executes the constructor described above using the maximum calorie selection (defined as
298 in line 158) and the file name for the input data (shown in Figure 15.7). In line 163, the method
named fit is invoked, which starts the execution of GA as explained above.
15.6 Python Script for Dietary Menu Selection Using a GA
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
# The City College of New York, City University of New York
# Written by Olga Chsherbakova
# August, 2023
#
#
#
#
#
#
#
KNAPSACK PROBLEM DESCRIPTION:
Given a knapsack with a maximum capacity of W and a set of items
of varying weights and values, select the items to place into the
knapsack such that the total value of the selected items is maximized
without exceeding the weight limit W.
from deap import base, creator, tools, algorithms
import random
import pandas as pd
import numpy as np
# class provides methods for setting up and running the genetic algorithm
# for solving the knapsack problem with given constraints.
class GeneticOptimizerKP:
# Initialization
# The __init__ method takes the calorie_capacity and items_file as
# parameters. It initializes attributes for storing parameters, data and
# tool configurations.
def __init__(self, calorie_capacity, items_file):
self.calorie_capacity = calorie_capacity
self.items_file = items_file
self.items_df = None
self.chromosome_length = None
self.A = None
self.b = None
self.toolbox = None
self.best_chromosome = None
# use del statement to remove previously created individuals (if any)
if hasattr(creator, "Fitness"):
del creator.Fitness
if hasattr(creator, "Individual"):
del creator.Individual
def _init_toolbox(self):
# Create a Fitness class inheriting with a single weight (1.0)
creator.create("Fitness", base.Fitness, weights=(1.0,))
# Create an Individual class inheriting from list with a
# fitness attribute
creator.create("Individual", list, fitness=creator.Fitness)
# register functions needed for GA
self.toolbox = base.Toolbox()
#generate a random integer (either 0 or 1) and
#represents a possible value for a gene in the chromosome.
self.toolbox.register("attr_bool", random.randint, 0, 1)
#function is used to create an individual for the genetic algorithm.
self.toolbox.register("individual",
#initializes a sequence with repeated values generated by
# another function.
tools.initRepeat,
# create an individual for the class
creator.Individual,
# generate the values for the codons of an individual
self.toolbox.attr_bool,
# specify number of codons an individual has, based on the
# chromosome_length
n=self.chromosome_length)
# create a population of individuals for the genetic algorithm
self.toolbox.register("population", tools.initRepeat,
list, self.toolbox.individual)
# evaluate the fitness of an individual
self.toolbox.register("evaluate", self.fit_func)
# adds a penalty to the fitness score of individuals that
# does not satisfy the constraint.
243
244
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
15 GAs for Dietary Menu Selection
self.toolbox.decorate("evaluate",
tools.DeltaPenalty(self.validate, -1.0))
# registers a crossover function
self.toolbox.register("mate", tools.cxOnePoint)
# performs bit-flip mutation. The indpb parameter specifies
# the probability of each bit in the chromosome being flipped.
self.toolbox.register("mutate", tools.mutFlipBit, indpb=0.01)
# function from deap, performs tournament selection.
# tournsize specifies the number of individuals for a tournament.
self.toolbox.register("select", tools.selTournament, tournsize=5)
# fit_func is the fitness function that will be minimized by GA.
# fit_func returns the negative of the sum of the value
# in fit_func: input is the chromosome,
# funcion: -(chromosome * items_table.value)
def fit_func(self, chromosome):
return np.dot(chromosome, self.items_df['Protein']),
# validate function receives a chromosome as an input and returns True
# if it is a valid chromosome, otherwise it returns False
#
# validity is tested with the inequality: A*chromosome <= b
def validate(self, chromosome):
return np.dot(self.A, chromosome) <= self.b
# chromosome for the GA will be a vector of binary digits, where
# each bit represents the selection of a paticular item:
#
- 0 if the item is NOT selected
#
- 1 if the item is selected
# chromosome length is the number of items in iventory
# example:
#
[1 0 1 0 0 0 0 0 1 1] - this chromosome represents a set of 10 items
# out of which four items are selected
def fit(self):
# load the items from a csv file and store it into a table
self.items_df = pd.read_csv(self.items_file)
# set the length of the chromosome to the number of items
self.chromosome_length = len(self.items_df)
# A and b are used to apply the following constraint on GA:
# A*chromosome <= b
# Set A to the vector of calories and b to the calorie capacity:
self.A = self.items_df['Calories']
self.b = self.calorie_capacity
self._init_toolbox()
# create an initial population with 20 individuals
pop = self.toolbox.population(n=20)
# HallOfFrame class is used to store the best individuals found during
# the evolution process. Creating it with a capacity of 1,
# which means it will store the single best individual.
self.best_chromosome = tools.HallOfFame(1)
# cxpb=0.7: The prob of performing crossover between individuals,
# which means there is a 70% chance to perform crossover.
# mutpb=0.2: The prob of performing mutation for each codon,
# which means there is a 20% chance of mutation for each codon.
# ngen=50: The number of generations or iterations for which the
# algorithm will run.
# HallOfFame object where the best individuals will be stored.
# verbose=False: This parameter determines whether the algorithm
# progress details are displayed.
algorithms.eaSimple(pop, self.toolbox, cxpb=0.7, mutpb=0.2, ngen=50,
halloffame=self.best_chromosome, verbose=False)
# the best individual from best_chromosome[0] to retrieve which
# represents the best solution that the algorithm converged to.
if self.validate(self.best_chromosome[0]):
selection = self.best_chromosome[0]
else: # if not valid, make all choices zero:
15.7 Matlab Script for Dietary Menu Selection Using a GA
(a)
(b)
Figure 15.8 Sample runs of Python script presented in Section 15.6: (a) 277 total calories and 32 grams of
protein, and (b) 297 total calories and 48 grams of protein.
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
selection = [0] * self.chromosome_length
selection_df = self.items_df[np.array(selection, dtype=bool)]
print('******************** GA FINISHED ********************')
print("OPTIMAL SELECTION OF ITEMS: {}".format(
list(selection_df['Name'])))
print("TOTAL PROTEIN VALUE OF SELECTION: {}" .format(
selection_df['Protein'].sum()))
print("TOTAL CALORIE OF SELECTION: {}" .format(
selection_df['Calories'].sum()))
calorie_capacity = 298 # user input; max calorie carried by knapsack;
items_file = 'items2.csv' #user input; data file
optimizer = GeneticOptimizerKP(calorie_capacity, items_file)
print(f'*********** READING ITEMS FROM {items_file} ************')
print(f'CALORIE LIMIT IS SET TO {calorie_capacity}')
optimizer.fit()
Two sample runs of the Python script presented above generate two different solutions, one with
277 total calories and 32 grams of protein and the other with 297 total calories and 48 grams of
protein, as shown in Figures 15.8a and b, respectively. Since GA is a nondeterministic algorithm,
multiple runs with the same input data may generate different, but equally valid, results.
15.7 Matlab Script for Dietary Menu Selection Using a GA
Matlab has a built-in function to implement a GA, called ga, which requires several options to be
defined as shown in Figure 15.9. In the Matlab implementation, each chromosome in a population
must satisfy an inequality defined as
A ⋅ chromosome† ≤ b†
(15.3)
where, for this example, A is a vector containing calorie values, b is the calorie upper limit and
† represents the matrix transpose operator. If this inequality is not satisfied, Matlab declares the
individual invalid and discards it.
As can be seen in Figure 15.9, the parameters of ga include the name of the fitness function, chromosome length, linear inequality constraints (i.e. so-called A and b matrices), non-linear inequality
245
246
15 GAs for Dietary Menu Selection
Figure 15.9
Parameters for GA implementation in Matlab.
constraints (if any), lower and upper bounds for codon values in a chromosome, non-linear constraints (if any), locations of integer codons in a chromosome, and additional options such as
population size, crossover function definition and the maximum number of generations. At completion, ga returns the resulting population array from the last generation where the top performing
candidates are placed at the top of the array.
A sample Matlab script implementing the KP using a GA is presented below. The sugar capacity is
defined in line 13. At line 40, the fitness function is defined as the multiplication of the chromosome
and the protein column of the food items table. The negative sign in line 40 indicates that it is a
maximization problem. At line 45, calorie values constitute the A vector, and at line 46 the calorie
capacity from line 13 is used as the b value needed for the ga function as defined in Eq. (15.3).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
% The City College of New York, City University of New York
% Written by Ricardo Valdez and Kelvin Ma
% August, 2020
%
% KNAPSACK PROBLEM DESCRIPTION:
% Given a knapsack (bag) with a maximum capacity of W and a set of items
% of varying weights and value, select items to put place into the
% knapsack such that the total value of the selected items is maximized
% without exceeding the weight limit W.
clc;
clear;
calorie_capacity = 290; % user input; max calorie carried by knapsack;
items_file = 'items.csv'; % user input; data file
% load the items from a csv file and store it into a table
items_table = readtable(items_file);
fprintf('*********** READING ITEMS FROM %s ************\n',items_file);
%fprintf('SUGAR CAPACITY IS SET TO %d\n',sugar_capacity);
%fprintf('CALORY CAPACITY IS SET TO %d\n', calory_capacity);
fprintf('CALORIE LIMIT IS SET TO %d\n', calorie_capacity);
% the chromosome for the genetic algorithm will be a vector of binary
% digits, where each bit represents the selection of a paticular item:
%
- 0 if the item is NOT selected
%
- 1 if the item is selected
%
chromosome length is the number of items in iventory
15.7 Matlab Script for Dietary Menu Selection Using a GA
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
% example:
%
[1 0 1 0 0 0 0 0 1 1] - this chromosome represents a set of 10 items
% out of which four items are selected
%
% set the length of the chromosome to the number of items
chromosome_length = height(items_table);
% fit_func is the fitness function that will be minimized by the genetic
% algorithm. fit_func returns the negative of the sum of the value
% in fit_func: input is the chromosome, body is what follows it:
% -(chromosome * items_table.value)
fit_func = @(chromosome) -(chromosome * items_table.Protein);
% A and b are used to used to apply the following constraint on the
% genetic algorithm: A*chromosome <= b
% We set A to the vector of calories and b to the calorie capacity
A = items_table.Calories'; % ' means tranposed
b = calorie_capacity;
% Lb is a vector that represents the lowerbounds for each gene.
% since all genes are binary digits, the lowerbounds will be set to 0
% put zeros into Lb[1][length]
Lb = zeros(1,chromosome_length);
% Ub is a vector that represents the upperbounds for each gene.
% since all genes are binary digits, the upperbounds will be set to 1
% put ones into Ub[1][length]
Ub = ones(1,chromosome_length);
% int_indices is an vector containing the indices of genes that are
% integers. since all genes are binary digits, all genes are integers.
% 1:chromosome_length creates a vector that increments from 1 to
% chromosome_length
int_indices = 1:chromosome_length;
% run the genetic algorithm
disp('******************** GA STARTING ********************');
% set options to: do not display
options = optimoptions('ga','display','off');
%options = optimoptions('ga','display','on'); % set to: display
% ga(fitness func, noof genes in chromosoem, A, b, A_eq, b_eq, Lb, Ub,...
% Non-lin contraint, indices for integer genes, additional options)
% vector of Aeq <= equations, vector of beq <= equations
% if integers are used in chromosome, they have to be left blank
selection = ga(fit_func,chromosome_length,A,b, ...
[],[],Lb,Ub,[],int_indices, options);
disp('******************** GA FINISHED ********************');
% print out the genetic algorithm result:
message = sprintf('OPTIMAL SELECTION OF ITEMS: [');
for i = 1:chromosome_length
if selection(i) == 1
message = sprintf('%s %s', message, string(items_table.Name(i)));
end
end
fprintf('%s]\n',message);
fprintf('TOTAL PROTEIN VALUE OF SELECTION: %d\n',...
selection * items_table.Protein);
fprintf('TOTAL CALORIE OF SELECTION: %d\n', ...
selection * items_table.Calories);
disp('*****************************************************')
Since GA is nondeterministic in nature, the results may differ for multiple runs of the same script
with the same input data, all of which satisfy the restrictions of the KP but make different selections.
In other words, while keeping the total number of calories to be less than or equal to the defined
limit, GA tries to maximize the total protein amount by the selections from the food table. Two
runs of the Matlab script presented above are shown in Figures 15.10a and b. In the first run, the
choice is egg, shrimp, swordfish with 287 calories and a protein total of 62 grams, whereas
247
248
15 GAs for Dietary Menu Selection
(a)
(b)
Figure 15.10 Sample runs of the Matlab script presented in Section 15.7 generating results with (a) 287
total calories and 62 grams of protein, and (b) 277 total calories and 62 grams of protein.
the second run chooses egg, shrimp, tomato, banana with 277 calories and 62 grams of
protein. Of course, both results are below the limit of 290 grams of sugar.
15.8 Concluding Remarks
A wide variety of real-life applications can be modeled as the KP or MKP class of problems. Modeling a real-life problem as a KP or MKP requires the experience and skill-set typically available
to computationally trained engineers. Because the KP and MKP are NP-complete problems, evolutionary algorithms such as GAs are required to obtain solutions in linear time. When employing
off-the-shelf software packages such as Matlab, Python or similar software, one has to be careful
about the assumptions used, not typically stated, in the implementations since they may restrain
evolutionary searches in exploring large solution spaces. For this reason, it may be necessary to
implement special-purpose software custom-tailored to handling computationally expensive problems with large sets of data to control mutation probabilities, number of generations and stopping
criteria for the GA.
15.9 Exercises for Chapter 15
1) Shopping planning: Given a list of items to buy as shown in Figure 15.11a, Roberta has a shopping trolley that can carry up to ten kilos of clothes. Roberta is at a market trying to buy as many
clothes as possible while staying under 10 kilos. Write a Matlab script similar to the one given
in Section 15.7 to find an acceptable solution for Roberta.
2) Air conditioner selection: Given the list of air conditioner manufacturers in Figure 15.11b, the
Schon family is planning to buy two units for their home. They want to make sure that the units
15.9 Exercises for Chapter 15
(a)
(b)
(c)
(d)
Figure 15.11 KP exercises: (a) information for shopping planning, (b) information for AC selection,
(c) information for course selection, and (d) information for public budget planning.
can run simultaneously and not cause a power outage in their home (maximum 24 kW). Write
a Matlab script similar to the one given in Section 15.7 to find an acceptable solution for the
Schon family.
3) Course selection: Given a list of classes at the City College as shown in Figure 15.11(c), Stanley
wants to pick as many classes as he can as long as the selection is below 18 credits to finish his
computer engineering degree early. Write a Matlab script similar to the one given in Section 15.7
to help Stanley in his selection of courses.
4) Public budget planning: The City of Paris is trying to pick a combination of buses (at most one
from each category) from the buses shown in Figure 15.11d. The goal is to maximize the total
number of passengers while staying within the limits on budget (maximum of 2 800 000 euros).
Write a Python script similar to the one given in Section 15.6 find an acceptable solution for
Paris.
249
251
16
GAs for Drone Flight Control
16.1 Introduction
In this chapter, we introduce a popular application of GAs to a real-life problem, namely flight
control of a drone that operates as a member of a swarm. Typically, a drone is capable of making
its own flight decisions. In this example application we assume that the goal for each drone is to
spread uniformly over an area of interest such that each drone positions itself as far away from
its near neighbours as its communication range allows. This task is especially formidable when
there are no mechanisms for synchronization, collaboration or a leader–follower scheme among
the drones. Drone flight control is selected for this chapter to illustrate the power of GAs in handling
these otherwise intractable problems.
16.2 UAV Swarms
Unmanned aerial vehicles (UAVs), commonly referred to as drones, are capable of eliminating the need for deploying costly equipment and dispatching personnel to accomplish complex
objectives in many engineering applications. A swarm of UAVs can be rapidly sent into areas of
interest to operate as a group without preexisting infrastructural support. UAVs can form and maintain a mobile ad-hoc network (MANET) to accomplish tasks that are difficult with traditional
communication infrastructures with centralized controllers, human operators and synchronization among UAVs [18]. Typical examples include unmanned aerial patrols being deployed over
industrial sites to detect harmful gas leakages, checking agricultural fields for crop health, early
detection of forest fires, maintaining a temporary communication infrastructure (constellation of
access points and base stations) for ground entities such as fire-fighters in a disaster scenario, or
localization of radio sources in rescue missions [19].
A UAV is typically considered small if its weight is less than 2 kg and its wingspan is under
one metre. Using small, cheap and disposable UAVs, swarms with tens or even hundreds of
members can be formed. It is possible to launch them from different areas, such as a stationary
ground (e.g. from a base station), moving ground (e.g. from a truck) or airborne platforms (e.g.
a helicopter). These types of small swarms are especially useful for operations in hard-to-reach
locations. Despite their advantages, small UAVs may have inherent restrictions imposed by their
limited power supply and payload capabilities that restrict their usage to short-duration flights.
They also have limited on-board processing capabilities, which requires lightweight yet fast flight
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
252
16 GAs for Drone Flight Control
(a)
(b)
Figure 16.1 Examples of swarms by different-sized drones: (a) swarm of 103 small drones launched from
an airborne platform [117]/Military Times, and (b) swarm of three large drones each capable of 15 kg of
payload; Background – Peter Griffin/Public Domain Pictures.
control algorithms. In the example shown in Figure 16.1a, a swarm of more than 100 small
drones were successfully deployed from military planes during a set of experiments conducted
by the US Department of Defense. After being launched from an airborne platform, autonomous
3D-printed Perdix UAVs communicated with each other to accomplish missions of hovering over
a target and forming a circle in the sky [117].
Larger UAVs have a wide range of weights and sizes. A large drone may weigh as much as a
ton and have a wingspan of over 10 metres. They have advantages over smaller versions by being
able to carry heavier payloads and fly farther distances for longer durations. However, compared
to their smaller counterparts, they also tend to be more expensive to operate and require much
more extensive infrastructure to maintain. Since risks associated with potential crashing of large
UAVs could be catastrophic, they tend to be pilot-operated throughout their flights. Therefore, large
UAVs are typically used in forming small swarms. Agricultural tasks, such as crop dusting and field
watering, are among applications often assigned to large drones. Figure 16.1b shows a swarm of
large drones, where each UAV is capable of carrying 10-litre tanks for weed-controlling substances
and can spray tens of acres per hour. Swarms of large UAVs are mostly used in rural areas and are
operated by human pilots with limited autonomy [118].
Small UAV swarms are much more widely used due to their operational flexibility and with low
maintenance costs. In the rest of this chapter, without loss of generality, we will focus on small
UAVs operating as swarms using MANETs as a means of communication. However, we should
note that the GA-based flight control algorithms introduced in this chapter can be implemented
equally effectively for large-UAV swarms. We will use the terms drone, node, mobile node
and UAV interchangeably throughout this chapter.
16.3 UAV Flight Control
Without loss of generality and for the sake of simplicity, all drones are assumed to be identical
in terms of their flight and communication capabilities. In this setting, all UAVs have the same
maximum communication range, can broadcast messages to their near neighbours within range,
aerial speed and manoeuvring capabilities are identical, and have on-board computers capable of
making flight decisions.
There are several factors that have to be considered when designing flight control algorithms
for autonomous UAVs. Limitations in power, flight duration and payload capacity are just a few of
16.4 A Simple GA for UAV Flight Control
many aspects that influence operation of a drone and the types of tasks that it can perform. Flight
control algorithms must be designed such that these hardware limitations are taken into account
to fully utilize UAV capabilities in a diverse set of operations.
Let us also consider the environmental conditions in which small UAV swarms commonly
operate. MANETs are typically deployed for applications with severe restrictions in communication, power and knowledge. Dynamically and rapidly changing operational conditions make
topology control tasks even more challenging for swarms of autonomous UAVs using MANETs:
Each UAV makes movement decisions independently and without humans.
The UAVs have limited power and range of sensing.
● Range of communication is limited such that each UAV can only communicate with its near
neighbours.
● There are no central controllers.
● The UAVs do not have global network knowledge since the topology changes continuously.
● Each UAV has to use only information in its locale for making movement decisions (e.g. obstacles
and terrain surface knowledge).
● The UAVs do not have any knowledge of the deployment terrain in advance.
●
●
A swarm of autonomous UAVs, an arrangement in which each drone makes independent flight
decisions and controls its own actions, is necessary to accomplish a broad range of applications.
Such drones are specifically designed to operate in dynamic environments. Among various flight
control strategies, GA-based techniques are good candidates to provide autonomous control for
each drone. If correctly implemented, GA-based flight control can provide real-time guidance for
tasks requiring independence and agility. Well-designed GAs can adapt to unpredictable situations
without the need for superfluous computations and high-speed coordination with other swarm
members.
16.4 A Simple GA for UAV Flight Control
Let us now present a sketch for a GA-based flight control algorithm that can guide an autonomous
UAV. Obtaining a good fitness function is crucial for a GA to adequately control the actions of
autonomous UAVs. A computationally efficient fitness function should be defined to closely correlate flight and assignment goals. Carefully defined chromosome structure and fitness function(s)
are essential for a GA-based flight control algorithm to converge to a good solution. In our context,
convergence means that UAVs are spread uniformly over an area of interest. Coverage of operational area by UAVs approaches a maximum if the UAVs end up being separated from each other
as far away as their communication range allows. Crossover and mutation operations can greatly
affect computation time to find a satisfactory solution while eliminating unnecessary movements
of the UAVs [18, 19].
Let a chromosome represent a candidate position for a UAV to move to at the next time point.
In the pseudocode shown in Figure 16.2, the GA starts by generating a population of individuals
(i.e. a set of candidate positions to move). For our example, each candidate position is evaluated
using a fitness function, which considers the near-neighbour locations of the UAV. At each generation, the best parents are selected for a crossover operation to obtain the next (and hopefully
improved) set of candidate positions. This process is repeated throughout many generations until a
satisfactory next location is found or a predetermined number of generations is reached. The UAV
then moves to the best location found by the GA.
253
254
16 GAs for Drone Flight Control
Figure 16.2
A simple algorithm for GA-based UAV flight control.
16.4.1 Virtual Force-Based Fitness Function
Let us introduce a GA employing a fitness function based on virtual forces that are assumed to
be applied to a node by its neighbouring nodes. Let us refer to this GA as a force-based GA
(FGA) [18]. A node that resides inside the sphere centred at the location of node i with a radius
of communication range RCOM is called a near neighbour of node i. In implementing an FGA,
we propose that a virtual (i.e. imaginary) force is inflicted on node i by each of its near neighbours.
This virtual force is inversely proportional to the distance between node i and a near neighbour.
The total virtual force inflicted on node i, called Fi , is the summation of the forces applied to i by
all of its near neighbours. The FGA uses the total virtual force on a mobile node as the fitness value
for this position. If the total virtual force is smaller for a location j than for k, we conclude that j
is a fitter position than k. Therefore, for a set of candidate positions, the FGA fitness function
prefers a location at which the least amount of total force is applied to the given node.
Suppose mobile node i has two neighbours within its communication range, namely, nodes j
and k. A virtual (i.e. imaginary) force RCOM inflicted on node i by its two near neighbours j and k
can be formulated as follows:
(√
)
(xi − xj )2 + (yi − yj )2
)
(√
2
2
+ RCOM −
(xi − xk ) + (yi − yk ) ,
Fi = fj→i + fk→i = RCOM −
(16.1)
where fj→i and fk→i represent the forces applied to node i by near neighbours j and k, respectively,
and xv and yv are the coordinates of node v (for v = i, j, k). As can be seen in Eqn. (16.1), Fi
becomes 0 when each near neighbour is RCOM units away from node i, which is the best value
of the fitness. Therefore, for the general case of a given node i with n near neighbours, if all neighbours are RCOM units apart from node i, the best fitness is obtained. When all nodes independently
obtain this fitness, a uniform distribution of nodes will be obtained.
16.4 A Simple GA for UAV Flight Control
Figure 16.3 Node i and its two near
neighbours, namely nodes j and k; node m is too
far from i (i.e. outside the sphere centred on
node i’s position with a radius of RCOM ) and,
therefore, has no impact on the fitness
calculation for node i.
An FGA may use other fitness functions, such as Voronoi tessellations [119] or Yao diagrams
[120]. For example, when employing Voronoi tessellations, a candidate position’s distance from
the geometric centre of its tessellation may be used as its fitness value [119].
Let us now present a visualization of FGA fitness function computation. In Figure 16.3,
we observe that node i has two near neighbours, namely, nodes j and k. Node m, on the other
hand, is too far from i (i.e. it is outside the sphere centred at node i with a radius of RCOM ) and,
therefore, has no impact on the fitness calculations for node i. Note that node m will be considered
in the fitness computation for node k since the distance from k to m is less than RCOM (i.e. they are
near neighbours).
16.4.2 FGA Progression
Progression of the FGA in a given node takes place as follows. Initially, for a given node a and a
population size of N, the FGA considers N randomly chosen positions as the candidate locations
for node a to move to (i.e. forming the initial population):
0
P00 (a), P10 (a), · · · , P(N−1)
(a),
g
where Pj (a) represents individual j (i.e. the candidate position j) for node a in generation g (g = 0
for the initial population). Selecting the initial population randomly is one of the strengths of GAs
since a wide range of initial positions can be considered in parallel. Success of many traditional
optimization algorithms heavily depends on the quality of the initial conditions, which implies that
the user must be familiar with the solution space to make a good initial guess. This is a drawback of
traditional approaches since it is difficult to find suitable initial conditions for complex optimization
problems even if one is an expert in that particular subject.
For the general case, the fitness of a candidate position pC is computed as
{∑n
if n ≥ 1,
i=0 (RCOM − di )
fitness(pC ) =
(16.2)
∞
otherwise,
255
256
16 GAs for Drone Flight Control
where n is the number of near neighbours of candidate position pC . The fitness is computed using
the total distance of pC to its near neighbours, as long as it has at least one near neighbour. If pC
does not have any near neighbours, it means that mobile node will be disconnected from the other
nodes if it moves to position pC . In this case, a large integer value is assigned as its fitness as a penalty
against further consideration of candidate position pC . Although Eq. (16.2) tries to eliminate candidate positions that are isolated, a node may still be disconnected from a swarm after it moves
(i.e. swarm may be partitioned) due to other operational conditions. For example, if the mobile
nodes are moving much faster than the frequency at which the FGA is updating its movement decisions, it may be possible that, by the time an FGA computes a new direction, its neighbours are no
longer at their positions, and the swarm may be partitioned by mistake. Finding a balance between
the frequency of running the FGA and the node speed is an empirical problem to be solved by engineers operating in a specific field. As the FGA proceeds for node a from generation g to (g + 1),
it performs the following genetic operations:
evaluate the fitness of each individual (i.e. candidate position)
sort the individuals based on their fitness from best to worst
● select parents giving preference to better performing individuals in generation g (the so-called
elitist method)
● perform crossover to generate offspring from the selected parents
● perform mutation on offspring based on a mutation probability
● evaluate the fitness of offspring
● replace less fit individuals in generation g with better-performing offspring (if any) and form
generation (g + 1) as follows:
●
●
(g+1)
P0
●
(g+1)
(a), P1
(g+1)
(a), · · · , P(N−1) (a)
repeat the above steps until a stopping criterion is satisfied
As the number of generations increases, the FGA finds more suitable positions for a mobile
node to eventually move to. However, one must identify a stopping criterion for the FGA to avoid
oscillations and possibly infinite loops. This criterion is highly dependent on the application
requirements. A few examples can be listed as follows:
a predetermined number of generations or a time limit has been reached
fitness of the best individual has reached a satisfactory value
● fitness of the best individual has not improved for several generations
● average fitness of the population has reached a satisfactory value
● average fitness of the population has not improved for several generations
●
●
After a node stops running the FGA, it moves to the best location indicated by the individual with
the best fitness in the last generation. Figure 16.4 illustrates a hypothetical case, where stars indicate
the candidate positions identified by the FGA for node i to move to. We can see in Figure 16.4a
that, among the six candidate positions, some are preferable to others. For example, consider the
candidate position located between nodes j and k. If node i moves to this position, it will be too close
to nodes j and k. Similarly, the candidate position located at the far upper right is not ideal since
moving there will most likely cause node i to lose communication with nodes j and k (distance will
be greater than RCOM ). If the candidate positions shown in Figure 16.4a were the final population
generated by the FGA, then node i will have to select one of the six positions and move there.
The best fitness among these six hypothetical positions should be a position that will move node i
away from nodes j and k, but not so far away that it will lose communication. Figure 16.4b shows
16.4 A Simple GA for UAV Flight Control
(a)
(b)
Figure 16.4 FGA operation for node i: (a) example candidate positions for node i to move as suggested by
the FGA are shown as stars (the FGA does not consider node m since it is too far from node i) and (b) node i
moves to the best candidate position.
a possible best candidate position that node i will move to. Once node i moves to this position, one
can see that fj→i and fk→i will be less (i.e. better) than what they were before it moved, as was shown
in Figure 16.3a.
In real-life swarms, it is not feasible to establish coordination or synchronization among nodes
since conditions change dynamically and rapidly due to movement. The FGA is periodically and
independently run by each node as they find better positions as all nodes move continuously.
The frequency of running the FGA depends on the specific application and its operational
characteristics.
16.4.3 Chromosome for FGA
In GAs, the role of a chromosome is to generate candidate solutions, which will be evaluated by a
fitness function to be considered for inclusion in the population. The chromosome is a numerical
representation of the parameters of the optimization problem at hand. Since the FGA aims to spread
a swarm of autonomous mobile nodes uniformly over an area of interest, its chromosome must
generate candidate positions for a mobile node to move to in the next time step. In our simple
example, if we assume that the speed of a mobile node is constant, then the chromosome represents
a candidate direction in which a mobile node will move.
Let us now define a chromosome that can be used in an FGA to guide mobile swarm nodes.
We will employ a binary chromosome, where the codons are either 0 or 1. In our example,
we assume that, in a two-dimensional space, a mobile node may only move in eight different
directions, namely east, west, north, south, north-east, north-west, south-east and south-west.
Three bits are needed to represent the eight possible directions in which the UAV can move. It is
clear that for a higher granularity of movements (e.g. 16, 32 or more positions), additional bits will
be needed in the chromosome. To accommodate three-dimensional node movement, we will need
additional bits in the chromosome. If we only consider moving in three possible vertical directions,
namely up, down and the same altitude, we will need two more bits in the chromosome.
257
258
16 GAs for Drone Flight Control
(a)
(b)
Figure 16.5 Chromosome for UAV movement: (a) the most significant two bits for vertical movement, and
(b) the least significant three bits for horizontal movement.
Many structures are possible for the arrangement of these five bits. In this example, we choose
the most significant two bits to represent the vertical movements of the UAV, whereas the least
significant three bits are used for the horizontal movement. We also decide that if the most
significant two bits are zeros, the UAV will not move, regardless of the values of its remaining three
bits. Figures 16.5a and b illustrate chromosome bit assignments for node movement directions.
Enumeration of two-dimensional movement directions for each possible value of the least
significant three bits of a chromosome are displayed in Figure 16.6. For example, a chromosome
Figure 16.6
Enumeration of two-dimensional movement directions by a chromosome.
16.4 A Simple GA for UAV Flight Control
Figure 16.7
Displacement vectors - examples in 2D movement.
of [x, x, 1, 1, 1] means that the node will move in a north-west direction, whereas [x, x, 1, 0, 0]
translates to a move south. The most significant two bits are marked as x and are not considered yet.
Displacement for a mobile node by a given chromosome is explained by the two examples presented in Figure 16.7. In Example 1, the chromosome is given as [x, x, 0, 0, 0], which implies that
the node will move north. Since the speed is the displacement of the node in unit time, we then
know that, after the move suggested by this chromosome, the new coordinates of the node will
be increased in the y dimension (i.e. north) by the value of the speed, whereas its coordinate in
the x dimension will stay the same. After this move, the node will end up at the position depicted
by a star. Similarly, in Example 2 of Figure 16.7, a chromosome of [x, x, 1, 0, 1] is considered for
the south-west direction, which means the initial coordinates will be decreased by the value of the
speed in both x and y dimensions. The new node position indicated by this chromosome is shown
as a star in Figure 16.7.
In Figure 16.8, coordinates of candidate positions are presented for all possible chromosome values of the least significant three bits for a node located at the (x0 , y0 ) position. For example, if the
Figure 16.8 Node displacement for each possible chromosome value: computation of candidate position
coordinates for a node located at (x0 , y0 ) position.
259
260
16 GAs for Drone Flight Control
chromosome is [x, x, 0, 0, 0], the node will move north from the (x0 , y0 ) position to (x0 , y0 + s),
where s is the speed. Similarly, a chromosome of [x, x, 1, 1, 1] implies that the node will move
north-east from position (x0 , y0 ) to a position with coordinates of (x0 − √s , y0 + √s ).
2
2
16.5 Python Script for UAV Flight Control Using a GA
In this section we present a sample implementation of a Python script for UAV flight control.
Because this Python script runs on a single-board computer, we assumed that a swarm operation
with multiple, independent and autonomous UAVs can be simulated as follows. A UAV runs an
FGA using its near neighbour information to compute a new position, but it will not move there
until all other UAVs complete their own computations. After all UAVs compute their next locations
to move, we assume they all move to these new locations simultaneously. This will constitute the
first step of the simulation experiment. Then, again one by one they run the FGA to compute the
next position based on new neighbour information and will move to their respective new positions
at the end of the second step of the simulation experiment. The number of steps is a variable defined
by the user.
This lock-step implementation without any inter-UAV communication is a rough approximation
of the parallel and independent computation that takes place in real-life drones. However, it demonstrates that the FGA may be a promising candidate in providing a uniform distribution of swarm
members. More advanced readers are encouraged to implement an FGA using threads [121]
and separate near neighbour tables for each UAV, which will be one step closer to a more realistic
implementation.
In lines 21–26, the number of simulation steps, speed, communication range, number of UAVs,
population size and number of generations are defined. In lines 33–41, initial UAV positions are
assigned randomly. Each altitude is increased by RCOM so that the UAVs do not crash into the
ground. In lines 93–97, displacement vectors are defined using the formulas given in Figure 16.8.
In lines 105–140, displacement is computed based on a given chromosome. In lines 142–154, the
function named get_neighbours finds the neighbours of a UAV based on indices of a UAV and
the current simulation step. In lines 156–168, the function named fit_func computes a fitness
score for a given position and the positions of the neighbours. In lines 170–193, a function named
animate creates a rudimentary animation of UAV positions for each step of the simulation.
In lines 206–212, GA attributes are registered into the toolbox library, including codons, chromosomes, population, crossover, mutation and selection operations.
In lines 214 to 240, a main function is introduced, which calls the functions described above.
Crossover and mutation probabilities are defined in line 218. The loop starting in line 219 iterates the number of simulation steps. The loop starting in 221 iteration for the number of UAVs.
Therefore, inside these two loops, UAV[i][j] represents a UAV with id j at time step i. UAV
neighbours are set at line 225 by calling the get_neighbours function. After the loops complete,
the function animate is called with UAV positions for all steps of the simulation.
1
2
3
4
5
6
7
8
# The City College of New York, City University of New York
# Written by Olga Chsherbakova
# September, 2023
#
#
#
# UAV Control with Genetic Algorithm
# Given the number of UAVs in a swarm and communication range, uniformly
16.5 Python Script for UAV Flight Control Using a GA
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
# spread the swarm such that they retain connectivity with neighbours
# while maintaining a distance close to radius from all neighbours
from deap import base, creator, tools, algorithms
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from mpl_toolkits.mplot3d import Axes3D
import math
import numpy as np
import random
n_UAV = 10 # 20 total moves for each UAV to make
speed = 1 # constant speed of each UAV
rcom = 10 # radius of communication
n_times = 10 # number of UAVs = 20
solutions_per_pop = 10
num_generations = 50
# Give each UAV a random starting location within half rcom of 0
# we multiply by rcom/sqrt(2) so that the distance from (0,0,rcom)
# is always less than rcom.
# UAV_positions is a 2-d matrix: [uav_no][x,y,z]
UAV_positions = np.zeros(shape= (n_times + 1, n_UAV, 3))
for x in range (0, n_UAV):
for y in range (0, 3):
UAV_positions[0][x][y] = (rcom/math.sqrt(2)) * random.random()
# we will assume that we want the UAVs to begin GA at a altitude
# higher than rcom (this makes it less likely to crash in the ground)
if (y%2 == 2):
UAV_positions[0][x][y] += rcom
print("Starting Coordinates: ")
for i in range(0, n_UAV):
print("UAV #{}:\t{}".format(i, UAV_positions[0][i]))
# Chromosomes
# The chromosome will represent the direction for the UAV to move
# Each chromosome will be represented as a vector of length 5
# Where the most significant 2 bits will represent altitude as such:
# [0, 0, x, x, x] = Stationary
# [0, 1, x, x, x] = Down
# [1, 0, x, x, x] = Up
# [1, 1, x, x, x] = Maintain current altitude
# There are 4 cardinal directions and 4 ordinal directions:
# North, Northeast, East, Southeast, South, Southwest, West, Northwest
# we can encode with the least significant 3 bits of the chromosome:
#
[x, x, 0,0,0] = North
[x, x, 0,0,1] = North-East
#
[x, x, 0,1,0] = East
[x, x, 0,1,1] = South-East
#
[x, x, 1,0,0] = South
[x, x, 1,0,1] = South-West
#
[x, x, 1,1,0] = West
[x, x, 1,1,1] = North-West
#
# Since there are no restrictions on the chromosomes other than length,
# there is no need for the A and b constraint matrices.
# The fitness function for these chromosomes are at the end of the script
chromosome_length = 5
# Displacement vectors on the X-Y plane
# Each direction and speed generate a displacement vector.
# The magnitude of this displacement vector is always equal to speed.
#
# Using the encodings for each direction, we can create an 2D array
# that holds the unit displacement vectors, then multiply all the unit
# vectors by speed. The array dimensions are
# displacement_vector[8][2] - 8 possible directions, 2 for x displacement
# and y displacement.
#
# for example, using the three least significant bits:
# (index) [unit displacement vector]
# 0
[0, 1]
- north
(xx000 is binary 0)
# 1
[1/sqrt(2), 1/sqrt(2)]
- northeast (xx001 is binary 1)
261
262
16 GAs for Drone Flight Control
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
# 2
# 3
# 4
# 5
# 6
# 7
[1, 0]
[1/sqrt(2), -1/sqrt(2)]
[0, -1]
[-1/sqrt(2)), -1/sqrt(2)]
[-1, 0]
[-1/sqrt(2)), 1/sqrt(2)]
- east
- southeast
- south
- southwest
- west
- northwest
(xx010 is binary 2)
(xx011 is binary 3)
(xx100 is binary 4)
(xx101 is binary 5)
(xx110 is binary 6)
(xx110 is binary 7)
# chromosome : b2b1b0 -> convert to decimal 0-7 and use as an index:
# [N, NE, E, SE, S, SW, W, NW]
# chromosome 101 -> 5 -> SW
displacement_vectors = np.array([[0, 1], [1/math.sqrt(2), 1/math.sqrt(2)],
[1, 0], [1/math.sqrt(2), -1/math.sqrt(2)],
[0, -1], [-1/math.sqrt(2), -1/math.sqrt(2)],
[-1, 0], [-1/math.sqrt(2), 1/math.sqrt(2)]])
displacement_vectors = displacement_vectors * speed
# The calc_distance calculates distance between two points in 3D space.
def calc_distance(old_coordinates, new_coordinates):
return math.sqrt((new_coordinates[0] - old_coordinates[0])**2 + \
(new_coordinates[1] - old_coordinates[1])**2 + \
(new_coordinates[2] - old_coordinates[2])**2)
# The decode function appears to be related to decoding a chromosome into a
# displacement vector. It takes a chromosome as input, which be represented
# as a binary sequence, and decodes it into a 3D displacement vector.
def decode(chromosome):
# initializes a NumPy array called displacement to store the
# resulting displacement vector. It initializes it with three zeros,
# representing the (x, y, z) components of the displacement.
displacement = np.zeros((3,), dtype=int)
# If chromosome[0] is 0 and chromosome[1] is 0, it means there's no change
# in the z-direction, so displacement[2] remains 0.
if (chromosome[0] == 0 and chromosome[1] == 0):
return displacement
#If chromosome[0] is 0 and chromosome[1] is 1, it sets displacement[2]
# to 1 * speed, implying a positive change in the z-direction.
if (chromosome[0] == 0 and chromosome[1] == 1):
displacement[2] = 1 * speed
# If chromosome[0] is 1 and chromosome[1] is 0, it sets displacement[2]
# to -1 * speed, implying a negative change in the z-direction.
if (chromosome[0] == 1 and chromosome[1] == 0):
displacement[2] = -1 * speed
#If chromosome[0] is 1 and chromosome[1] is 1, it sets displacement[2]
# to 0, which means that there is no change in the z-direction.
if (chromosome[0] == 1 and chromosome[1] == 1):
displacement[2] = 0
# calculates an index based on the values of three elements in the
# chromosome array to select a displacement vector from some predefined
# array based on the values of these elements.
LSB_index = 1 * chromosome[4] + 2 * chromosome[3] + 4 * chromosome[2]
# sets the displacement[0] and displacement[1] components of the
# displacement vector using values from array called displacement_vectors.
# The specific values used are determined by the LSB_index calculated
# in the previous step.
displacement[0], displacement[1] = \
displacement_vectors[LSB_index][0], displacement_vectors[LSB_index][1]
return displacement
# This function finds neighbouring UAVs for a given UAV at a specific time.
def get_neighbours(time, UAV_index):
neighbours = []
for i in range(0, n_UAV):
UAV = UAV_positions[time][i]
# If the distance between the current UAV and the target UAV is less
# than or equal to rcom, indicating that they are within communication
# range, the current UAV is considered a neighbour, and its position
# is appended to the neighbours list.
if (UAV_index != i and calc_distance(UAV,
UAV_positions[time][UAV_index]) <= rcom):
neighbours.append(UAV)
return neighbours
16.5 Python Script for UAV Flight Control Using a GA
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
# Calculate fitness score based on its displacement, position, and neighbours.
def fit_func(individual, current_UAV_coordinates, neighbours):
fitness_score = 0
displacement = decode(individual)
new_position = [current_UAV_coordinates[0] + displacement[0],
current_UAV_coordinates[1] + displacement[1],
current_UAV_coordinates[2] + displacement[2]]
if (new_position[2] < 0 ):
return [np.inf]
for neighbour in neighbours:
distance = calc_distance(neighbour, new_position)
fitness_score += (rcom - distance)
return [fitness_score]
def animate(UAV_positions):
# Create figure object.
fig = plt.figure()
# Create 3D axis object using add_subplot().
ax = fig.add_subplot(111, projection='3d')
u, v = np.mgrid[0:2*np.pi:10j, 0:np.pi:10j]
unit_sph_x = np.cos(u)*np.sin(v)
unit_sph_y = np.sin(u)*np.sin(v)
unit_sph_z = np.cos(v)
fig.show()
for i in range(0, n_times + 1):
ax.clear()
for j in range (0, n_UAV):
sph_x = (rcom*unit_sph_x + UAV_positions[i][j][0])
sph_y = (rcom*unit_sph_y + UAV_positions[i][j][1])
sph_z = (rcom*unit_sph_z + UAV_positions[i][j][2])
ax.plot_surface(sph_x,sph_y,sph_z, color='w', alpha = 0.05,
edgecolor = 'white')
ax.scatter(UAV_positions[i,:, 0], UAV_positions[i,:, 1],
UAV_positions[i,:, 2], s = 30, color = 'r' )
fig.canvas.draw()
plt.pause(0.25)
fig.canvas.flush_events()
plt.show()
if hasattr(creator, "Fitness"):
del creator.Fitness
if hasattr(creator, "Individual"):
del creator.Individual
INDIVIDUAL_SIZE = chromosome_length
creator.create("Fitness", base.Fitness, weights=(-1.0, ))
creator.create("Individual", list, fitness=creator.Fitness)
toolbox = base.Toolbox()
# Attribute generator
toolbox.register("attr_bool", random.randint, 0, 1)
toolbox.register("individual", tools.initRepeat, creator.Individual,
toolbox.attr_bool, n=INDIVIDUAL_SIZE)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)
toolbox.register("mate", tools.cxOnePoint)
toolbox.register("mutate", tools.mutFlipBit, indpb=0.01)
toolbox.register("select", tools.selTournament, tournsize=5)
def main():
# CXPB is the probability with which two individuals
#
are crossed
# MUTPB is the probability for mutating an individual
CXPB, MUTPB = 0.5, 0.2
for i in range(0, n_times):
print("\nTIME #{}\n".format(i))
for j in range(0, n_UAV):
print("\nRUNNING GA FOR UAV #{}\n".format(j))
UAV = UAV_positions[i][j]
print("CURRENT COORDINATES:\t{}\n".format(UAV))
neighbours = np.array(get_neighbours(i,j))
# print("neighbourS FOR UAV #{}\n{}".format(i, neighbours))
toolbox.register("evaluate", fit_func, current_UAV_coordinates =
263
264
228
229
230
231
232
233
234
235
236
237
238
16 GAs for Drone Flight Control
UAV, neighbours = neighbours )
pop = toolbox.population(solutions_per_pop)
hof = tools.HallOfFame(1)
pop, log = algorithms.eaSimple(pop, toolbox, CXPB, MUTPB,
num_generations, verbose= False, halloffame=hof)
new_coordinates = (UAV + decode(hof[0]))
print("NEW COORDINATES:\t{}\n".format(new_coordinates))
UAV_positions[i+1][j] = new_coordinates
animate(UAV_positions)
main()
(a)
(b)
Figure 16.9 Final positions of a swarm of 10 UAVs guided by the Python script given in Section 16.5 after
10 simulation steps: (a) good case, where all UAVs are connected as a swarm (using RCOM = 1 and speed = 1),
and (b) bad case, where some UAVs are disconnected from the swarm (using RCOM = 10 and speed = 10).
A swarm of 10 UAVs is simulated for 10 steps, where each UAV is guided by the FGA-based
flight control using the above Python scripts. Figure 16.9 shows the final positions of a swarm of
10 UAVs guided by the FGA implemented by the Python script given in Section 16.5. For a setup
with RCOM = 1 and speed = 1, the UAVs spread over a 3-D region while keeping the swarm connected, as shown in Figure 16.9a. However, when the UAV speed is increased tenfold, we observe
in Figure 16.9b that some of the UAVs become disconnected from the swarm and hence the swarm
is partitioned. This bad performance is most likely due to the inability of the FGA decisions to keep
up with the speedy topology changes stemming from the increased UAV speed.
16.6 Matlab Script for UAV Flight Control Using a GA
In this section, we present a sample Matlab script implementing UAV flight control. The Matlab
script uses the fitness function given in Eqs. (16.1) and (16.2), and the chromosome length and
codon meanings in terms of movement directions as in Section 16.4, with the simplifying assumption that all UAVs have the same speed and communication range. For ease of comparison for the
reader, the structure of the Matlab script is kept similar to the one in Python in Section 16.5. As in
the Python case, due to the single-processor environment where the script runs, each UAV makes
movement decisions one by one, and they then move to their new positions together. If threads
are used in script, a more realistic simulation can be constructed, allowing pseudo-parallel decision
16.6 Matlab Script for UAV Flight Control Using a GA
and movement capability for each UAV. Advanced readers are encouraged to convert the script
presented here to a version using threads.
In lines 14–20, the number of UAVs in the swarm, UAV speed, communication range and number
of moves (i.e. the length of simulation experiment) are defined. In lines 24–47, the UAV positions
are initialized randomly such that they are within RCOM distance of each other and with an altitude
higher than RCOM so that they will not crash into the ground during the simulation. Line 67 sets the
chromosome length to 5, with the same codon functionalities as defined in Section 16.4.3. Lines
68, 69 and 71 set up Matlab parameters for the GA, namely the upper and lower values for codons
and the indices used in chromosomes, respectively. Displacement vectors discussed in Figure 16.8
are defined in lines 97–101.
The simulation loop between lines 107 and 146 represents all actions to be taken by all UAVs
at each step of the experiment. An iteration of the loop, given between lines 111 and 145, corresponds to the actions taken by a single UAV. In lines 114 and 115, a near-neighbour list for this
UAV is obtained. Its current position is set in line 117. The fitness function for the FGA is called in
lines 122–124, which uses the following inputs: chromosome, current position, speed, neighbours,
communication range and displacement vectors. We will analyse the fitness function later as it is
defined in lines 168–192.
In lines 131 and 132, the GA is called, which returns the candidate positions into an array named
selection, whose top element is the best-performing candidate. This position is converted to
the coordinates of the next position for this UAV to move to in lines 135 and 137, then stored in the
UAV_Positions array to be used for the next simulation step. The loops for the number of UAVs
and animation steps complete in lines 144 and 145, respectively. After the positions of all UAVs
for all simulation experiment steps are computed, they are displayed using animation. For this
purpose, in line 149, a function named visualize_UAVs3D, is called with the following input
parameters: the positions of the UAVs at all steps, the number of simulation steps, the number of
UAVs and the communication range for a UAV. This function, as discussed further below, opens an
animation pane and displays movements of all UAVs for each step. In lines 150 to 162, the user is
asked if the animation is to be repeated. Otherwise, the script completes.
Between lines 168 and 192, a function named fitness_score is called with a chromosome,
current position, speed, an array of neighbours, RCOM and the displacement vectors. In line 173, a
function named candidate_pos is called, which is defined in lines 196–220 below. It computes
the next position based on the displacements shown in Figure 16.8 by using the bi2de function
for converting binary numbers to their equivalent decimals. For example in line 198, if the most
significant two bits are zeros, the next position of this UAV will be the same as its current position.
Similarly, in lines 203 to 206, if the most significant two bits are 01, the UAV will move up and the
displacement will be based on the remaining three bits.
In line 149, VisualizeUAVs3D is called, which displays UAV positions at every step in a simple animation. In lines 150–162, the user is asked either to replay the animation or exit the script.
In lines 163 to 192, a function named fitness_function is presented. In the loop presented
in lines 180–192, distance to the candidate position is computed for each neighbour. If this distance is less than RCOM , variable fitness_score is incremented by rcom minus this distance, as
presented in Eq. (16.2). For a candidate position that has no neighbours or is outside the communication range, a large integer is assigned as its fitness as a penalty so that it will not be considered
in any further computations.
In lines 193 to 222, using a chromosome, the next position is computed by the function named
get_next_position. Using the formulation presented in Figure 16.8, the node displacement
and the next position based on a chromosome value are computed.
265
266
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
16 GAs for Drone Flight Control
% The City College of New York, City University of New York
% Written by Kelvin Ma
% August, 2020
% UAV Control with Genetic Algorithm
% Given the number of UAVs in a swarm and a radius of communication,
% use a genetic algorithm to uniformly spread the swarm such that they
% retain connectivity with its neighbours while maintaining a distance
% close to radius from all its neighbours.
clc; clear;
% User inputs and initalizations
%
%user inputs:
noof_UAVs = 10; % number of UAVs = 20
% good: speed = 0.1* rcom: rcom = 10, speed = 1
% bad: when speed == rcom: rcom = 10, speed = 10
% ugly: speed = 0.8* rcom: rcom = 10, speed = 8
speed = 1; % constant speed of each UAV
rcom = 10; % radius of communication
noof_moves = 10; % 20 total moves for each UAV to make
% Create a 3D matrix to store all UAV positions - initalized all to 0's
% UAV_positions[noof_UAVs][3][noof_moves]
% 3 is for spatial (x, y, z) coordinates
UAV_positions = zeros(noof_UAVs, 3, noof_moves);
UAV_positions(:,:,1) = (rcom/sqrt(2)).*rand(noof_UAVs, 3);
% Give each UAV a random starting location within half rcom of 0
% rand function creates a random matrix of dimensions rand_mat[noof_UAVs][3]
% where each element is between 0 and 1, so we multiply by rcom/sqrt(2)
% so that the distance away from (0,0,rcom) is always less than rcom.
% UAV_positions is a 3-d matrix: [uav_no][x,y,z][time]
% Example: for positions of UAVs at time 1:
%
5.7610
1.1145
14.6368 <- UAV_1
%
6.4049
6.8631
10.2525 <- UAV_2
%
0.8979
6.7682
16.0043
%
6.4585
3.4321
16.6043
%
4.4715
5.6588
14.7994
%
0.6897
1.0033
15.3580
%
1.9693
2.9823
15.2547
%
3.8670
6.4752
12.7735
%
6.7706
5.6018
14.6349
%
6.8228
6.7846
11.2105 <- UAV_10
% we will assume that we want the UAVs to begin GA at a altitude higher
% than rcom (This makes it less likely to crash in the ground)
UAV_positions(:,3,1) = UAV_positions(:,3,1)+rcom;
% Chromosomes
% The chromosome will represent the direction for the UAV to move
% Each chromosome will be represented as a vector of length 5
% Where the most significant 2 bits will represent altitude as such:
% [0, 0, x, x, x] = Stationary
% [0, 1, x, x, x] = Down
% [1, 0, x, x, x] = Up
% [1, 1, x, x, x] = Maintain current altitude
% There are 4 cardinal directions and 4 ordinal directions - they are:
% North, Northeast, East, Southeast, South, Southwest, West, Northwest
% we can encode with the least significant 3 bits of the chromosome:
%
[x, x, 0,0,0] = North
[x, x, 0,0,1] = North-East
%
[x, x, 0,1,0] = East
[x, x, 0,1,1] = South-East
%
[x, x, 1,0,0] = South
[x, x, 1,0,1] = South-West
%
[x, x, 1,1,0] = West
[x, x, 1,1,1] = North-West
%
% Since there are no restrictions on the chromosomes other than length,
% there is no need for the A and b constraint matrices.
% The fitness function for these chromosomes are at the end of the script
chromosome_length = 5;
Lb = zeros(1,chromosome_length); % lower bound of gene values
Ub = ones(1,chromosome_length); % upper bound of gene values
% indices of the chromosome which are integers (all of them in this case)
int_indices = 1:chromosome_length;
% Displacement vectors on the X-Y plane
16.6 Matlab Script for UAV Flight Control Using a GA
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
% The chromosomes are the DIRECTION for the UAV to move. Each direction and
% speed generate a displacement vector in space for the UAV. The
% magnitude of this displacement vector is always equal to speed.
%
% Using the encodings for each direction, we can create an 2D array
% that holds the unit displacement vectors, then multiply all the unit
% vectors by speed. The array dimensions are
% displacement_vector[8][2] - 8 possible directions, 2 for x displacement
% and y displacement.
%
% for example, using the three least significant bits:
% (index) [unit displacement vector]
% 1=(0+1) [0, 1]
- north
(xx000 is binary 0)
% 2=(1+1) [1/sqrt(2), 1/sqrt(2)]
- northeast (xx001 is binary 1)
% 3=(2+1) [1, 0]
- east
(xx010 is binary 2)
% 4=(3+1) [1/sqrt(2), -1/sqrt(2)]
- southeast (xx011 is binary 3)
% 5=(4+1) [0, -1]
- south
(xx100 is binary 4)
% 6=(5+1) [-1/sqrt(2)), -1/sqrt(2)]
- southwest (xx101 is binary 5)
% 7=(6+1) [-1, 0]
- west
(xx110 is binary 6)
% 8=(7+1) [-1/sqrt(2)), 1/sqrt(2)]
- northwest (xx110 is binary 7)
% Note: the +1 is because MATLAB vectors start with index 1
% chromosome : b2b1b0 -> convert to decimal 0-7 and use as an index (+1):
% [N, NE, E, SE, S, SW, W, NW]
% chromosome 101 -> 5 -> 5+1 -> SW
displacement_vectors = [[0, 1]; [1/sqrt(2), 1/sqrt(2)]; ...
[1, 0]; [1/sqrt(2), -1/sqrt(2)]; ...
[0, -1]; [-1/sqrt(2), -1/sqrt(2)]; ...
[-1, 0]; [-1/sqrt(2), 1/sqrt(2)]];
displacement_vectors = speed .* displacement_vectors;
% Running the Genetic Algorithm for all UAV from time 0 to end
% set options to: do not display (change off to iter to display steps)
options = optimoptions('ga','display','off'); % set not to display
% Each UAV must run their own genetic algorithms at every time step
for t = 1:noof_moves-1
fprintf("\n\n***** TIME IS NOW: %d*****\n", t)
% iterate through the UAVs, the fitness function will change depending
% on the UAV number we are on
for UAV_num = 1:noof_UAVs
fprintf("***** BEGINNING GA FOR UAV #%d *****\n", UAV_num)
% get the neighbours of the UAV that is running GA - exclude itself
neighbours = UAV_positions(:,:,t);
neighbours(UAV_num,:) = [];
% get current position of UAV that is running ga
current_position = UAV_positions(UAV_num, :, t);
% create a function pointer for the fitness function
% (in Matlab we declare this inside the loop, o.w. does not update
% parameter values at each loop iteration)
fit_func = @(chromosome) fitness_function(chromosome, ...
current_position, speed, neighbours, rcom, ...
displacement_vectors);
% Run the genetic algorithm for the current node
% Matlab has its own default values for population size and number
% of generations
% selection contains the best chromosome returned by ga (it is the
% next direction to move):
selection = ga(fit_func,chromosome_length,[],[], ...
[],[],Lb,Ub,[],int_indices, options);
% calculate the next position based on selection from ga:
displacement=displacement_vectors(bi2de(selection(3:end),...
'left-msb')+1,:);
next_position = get_next_position(selection, current_position, ...
speed, displacement_vectors);
% keep a record of these positions for the next ga
UAV_positions(UAV_num,:,t+1) = next_position;
fprintf("\tselection fitness = %f\n", fit_func(selection))
fprintf("***** FINISHED GA FOR UAV #%d AT TIME %d *****\n", ...
UAV_num, t)
end
267
268
16 GAs for Drone Flight Control
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
end
% Visualize the UAVs
visualize_UAVs3D(UAV_positions, noof_moves, noof_UAVs, rcom);
open_anim = true;
while open_anim == true
fprintf("Replay the animation?\n\tenter 1 for yes\n\tenter 2 for no\n");
user_choice = input('');
if user_choice == 2
open_anim = false;
fprintf('Thank you. Have a nice day!\n');
break;
else
close all
visualize_UAVs3D(UAV_positions, noof_moves, noof_UAVs, rcom);
end
end
% Fitness function
% Functions in MATLAB which are multiple lines are defined at the bottom
% of the script.
% MATLAB will use GA to minimize the value variable called fitness_score
function fitness_score = fitness_function(chromosome, position, speed, ...
neighbours, rcom, displacement_vectors)
fitness_score = 0;
neighbour_count = 0;
% calculate the candidate position from the chromosome and displacement
% vectors
candidate_pos = get_next_position(chromosome, position, speed, ...
displacement_vectors);
% iterate through all the neighbours and calculate the fitness
% fitness += (rcom - distance from neighbour[i])
% ga minimizes this number, the closer to 0 the better
for i = 1:length(neighbours)
distance = norm(candidate_pos - neighbours(i,:));
if distance <= rcom
neighbour_count = neighbour_count + 1;
fitness_score = fitness_score + (rcom - distance);
end
end
% maximum penalty for positions with no neighbours or positions that
% crash to the ground
if neighbour_count < 1 || candidate_pos(3) <= rcom
fitness_score = abs(intmax);
end
end
% Getting the next position from the chromosome
% Functions in MATLAB which are multiple lines are defined at the bottom
% of the script.
% this function gets the next position from a chromosome
function next_pos = get_next_position(chromosome, position, speed, ...
displacement_vectors)
if bi2de(chromosome(1:2)) == 0
% stationary - next position = current_position
next_pos = position;
elseif bi2de(chromosome(1:2), 'left-msb') == 1
% UAV moves up
displacement = displacement_vectors(bi2de(chromosome(3:end), ...
'left-msb')+1,:);
next_pos(1:2)= position(1:2) + displacement;
next_pos(3) = position(3) + speed;
elseif bi2de(chromosome(1:2), 'left-msb') == 2
% UAV moves down
displacement = displacement_vectors(bi2de(chromosome(3:end), ...
'left-msb')+1,:);
next_pos(1:2)= position(1:2) + displacement;
next_pos(3) = position(3) - speed;
elseif bi2de(chromosome(1:2), 'left-msb') == 3
% UAV mainstains altitude
next_pos = position;
displacement = displacement_vectors(bi2de(chromosome(3:end), ...
'left-msb')+1,:);
16.6 Matlab Script for UAV Flight Control Using a GA
219
220
221
222
next_pos(1:2)= position(1:2) + displacement;
else
end
end
Below we present a function named VisualizeUAVs3D is presented, whose input parameters
are the UAV positions, the number of steps in the simulation experiment, the number of UAVs
and the communication range. In lines 5 to 25, it establishes the scales based on minimum and
maximum values of coordinates. In the loop presented between lines 27 and 51, for each move and
for each UAV, coordinates and a sphere around the UAV position are drawn.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
%% Visualizing the results
function visualize_UAVs3D(UAV_positions, noof_moves, noof_UAVs, rcom)
figure;
% initialize variables needed to draw a sphere
[unit_sph_x, unit_sph_y, unit_sph_z] = sphere;
min_x = min(min(UAV_positions(:,1,:))) - rcom;
max_x = max(max(UAV_positions(:,1,:))) + rcom;
min_y = min(min(UAV_positions(:,2,:))) - rcom;
max_y = max(max(UAV_positions(:,2,:))) + rcom;
min_z = 0;
max_z = max(max(UAV_positions(:,3,:))) + rcom;
x_span = (max_x - min_x);
y_span = (max_y - min_y);
x_axis_min = (round(min_x/rcom)-1)*rcom;
x_axis_max = (round(max_x/rcom)+1)*rcom;
y_axis_min = (round(min_y/rcom)-1)*rcom;
y_axis_max = (round(max_y/rcom)+1)*rcom;
z_axis_min = min_z;
z_axis_max = (round(max_z/rcom)+1)*rcom;
iconsize = [x_span/30 y_span/30];
% read image file for UAV icon
%img_name = 'UAV_icon.png';
img_name = 'UAV_icon2D.png';
UAV_icon = imread(img_name, 'png');
for t = 1:noof_moves
pause(0.25); % pause shortly for viewing
hold off
for UAV_num = 1:noof_UAVs
% get the x and y position of UAV
x_pos = UAV_positions(UAV_num,1,t);
y_pos = UAV_positions(UAV_num,2,t);
z_pos = UAV_positions(UAV_num,3,t);
sph_x = (rcom*unit_sph_x + x_pos);
sph_y = (rcom*unit_sph_y + y_pos);
sph_z = (rcom*unit_sph_z + z_pos);
surf(sph_x, sph_y, sph_z,'FaceColor', 'blue', ...
'LineStyle', '-', 'EdgeColor', 'blue', ...
'FaceAlpha', 0.05, 'EdgeAlpha', 0.1);
xlim([x_axis_min x_axis_max])
ylim([y_axis_min y_axis_max])
zlim([z_axis_min z_axis_max])
hold on
% put the UAV icon at the UAV's position and adjust opacity
scatter3(x_pos, y_pos, z_pos, 'filled');
end
plot_title = sprintf('UAV positions at t = %d', t);
title(plot_title);
end
end
In Figure 16.10a, the swarm is setup with RCOM = 1 and speed = 1. We can see that after 10
moves, all UAVs are within the swarm, and each UAV has one or more near neighbours. However,
269
270
16 GAs for Drone Flight Control
(a)
(b)
Figure 16.10 Final positions of a swarm of 10 UAVs guided by FGA-based flight control implemented by
the Matlab script presented in Section 16.6 after 10 simulation steps: (a) for RCOM = 1 and speed = 1, all
UAVs are within the swarm, and (b) for RCOM = 10 and speed = 10, the swarm is partitioned with some UAVs
disconnected from the swarm.
Figure 16.10b, a swarm operating with RCOM = 1 and speed = 10, where the UAV speed is increased
tenfold, shows that the swarm is partitioned with some UAVs being disconnected from the swarm.
In this case, the movement decisions made by the FGA-based flight control may be too slow and, by
the time a movement decision is made, neighbouring UAVs have already moved into different positions. In a realistic operating environment, there are always trade-offs among various parameters
of a swarm. In this case, we see that increasing UAV speed arbitrarily may partition the swarm.
16.7 Concluding Remarks
In this chapter, we introduced an example of a popular application, namely flight control of
autonomous drones operating as a swarm. In this example application, the goal for the swarm of
autonomous drones is set as spreading uniformly over a geographical area of interest, without
any intervention from a human pilot or central controller. We assumed that drones do not have
information about the swarm member positions except for the near neighbours within their
immediate communication range, and that there is no form of coordination or synchronization
among the drones. This example in fact can be considered as a simplified version of many
real-life drone applications such as rescue missions, environmental protection tasks, fire-fighting
operations, multi-vehicle transportation, and others.
We introduce a UAV flight control mechanism employing a GA, where a chromosome represents
the direction in which a UAV will move at the next step. A fitness function is defined where virtual
forces are imagined to be applied to a node by its immediate neighbours. The GA favours positions
that place a UAV away from its neighbours without losing communication with them. Candidate
positions that may isolate a UAV or move it too close to its neighbours will not have good fitness
values.
In Python and Matlab implementations of this flight control algorithm, we show that relatively
simple scripts can successfully realize a GA for a set of autonomous drones operating as a swarm.
In these simple scripts, we see that drone speed and communication range are the major variables
to prevent swarm partitioning.
16.8 Exercises for Chapter 16
If the computation load of flight control is kept light, bio-inspired algorithms such as GAs can be
perform in real-time (or near-real-time) flight control. If more conditions are included in a chromosome or a more advanced fitness function is employed, more powerful flight control algorithms may
be constructed. However, increased complexity implies that computational time for flight control
will most likely increase, which may prevent some applications from operating in real time.
It is possible that other biologically inspired algorithms could be used for flight control of
autonomous drones operating as a swarm. For example, one could easily adapt a similar flight control mechanism by replacing the GAs with particle swarm optimization or differential evolution
algorithms (see Sections 18.2 and 18.3 in Chapter 18).
16.8 Exercises for Chapter 16
1) Construct a chromosome for the FGA presented in Figure 16.5 that is capable of selecting 64
different directions for a UAV to move at each step. What are the advantages and disadvantages
of using this chromosome?
2) Propose a different chromosome for the FGA presented in Figure 16.5 using three codons, each
using decimal values. Are there any advantages of using this chromosome? Any disadvantages?
3) Propose a fitness function that will force the UAVs to stay in a straight line.
4) Modify the Python script given in Section 16.5 using threads. Run it several times with the same
parameter values. Do you see any difference between each run? Why do you think this happens?
5) Repeat Exercise 1 for the Matlab script given in Section 16.6 using threads.
271
273
17
GAs for Route Optimization
17.1 Introduction
In this chapter, we present another important application of GAs, namely the route optimization
problem, where a set of nodes interconnected with segments with different cost values are to be
covered with minimum cost. We can visualize this problem as follows: we are given a set of cities
and we need to visit all of them such that the total length of the route is a minimum. If the cost
is defined as the time to travel between cities, then the goal is to visit the cities in minimum time.
This problem is known in computer science as the travelling salesman problem (TSP),
where a salesman needs to visit each city only once, starting and returning to the same one [122].
Requirements that the salesman can visit each city only once and that the total length (or its equivalent cost) of the entire trip must be minimized make the travelling salesman class of problems a
formidable challenge. Their solution complexity grows as the factorial of the number of cities and
the connections between them.
Optimization problems for engineering tasks are often complex and frequently need to determine
the most effective ordering of a sequence of actions to accomplish a given set of goals. Consider a
delivery company that dispatches a fleet of hundreds of trucks to distribute packages to a set of
locations every morning. Each truck starts at a base location, makes multiple stops during the day
and returns to the base at the end of its shift. The goal for a manager is to ensure that each trip visits
as many delivery points as possible while travelling the shortest distance.
In another application, consider the motor vehicle manufacturing industry, which produces
more than 70 million cars every year globally. For each car, there are thousands of screws to
be tightened and thousands of spot welding operations to be performed. The order in which
the screws are to be tightened or the metal points welded can be formulated as a TSP. A robot
starts welding from a selected point and visits dozens of points in a given order until returning
to its initial position after completing the last spot weld. In this case, the robot and points to
be welded correspond to the salesman and the cities of the TSP, respectively. A spot welding or
screw tightening robot, if not programmed carefully, may waste time on each car and, hence, may
significantly impact factory output and operating budget. The savings for these applications is in
the tens of billions of dollars. Similarly, in the electronics industry, finding the best order to solder
components or pierce holes on printed circuit boards by robot are problems with high impact on
productivity. TSP solutions can minimize soldering time, path complexity, energy and material
consumption for any electronics manufacturer [123].
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
274
17 GAs for Route Optimization
In this chapter, we present a GA-based solution for the TSP which computes optimal routes
(although not necessarily the absolute minimum-cost ones) in linear time using GAs. We show
that this relatively simple transformation of the problem to the AI domain can be implemented
using the library functions in Python and Matlab.
17.2 Definition of the TSP
An example of the TSP is presented in Figure 17.1a, where each red dot must be visited once with
minimum total length of the cycle [124]. A possible solution is shown in Figure 17.1b. Another TSP
example and one possible solution are shown in Figure 17.2 [125]. There are worldwide competitions among researchers for measuring the effectiveness of their TSP solutions. Examples include
covering 24 978 cities and towns in Sweden and 15 112 in Germany (Figure 17.3) [124] and 3143
counties of the USA (a near-optimal length of 93 466 miles) (Figure 17.4) [126]. For these benchmarks, the length of the road between two cities is taken as the cost of the edge.
Another simple example of TSP is shown in Figure 17.5, where a traveller starting in Detroit
has to visit Chicago, Columbus and Indianapolis before returning to Detroit. The roads connecting
Figure 17.1 Example of the TSP:
(a) finding the shortest possible loop that
connects every grey dot only once, and
(b) one possible TSP solution (Adapted
from [127]).
(a)
(b)
(a)
(b)
Figure 17.2 Example of the TSP: (a) the goal is to find a loop that connects all dots by visiting each dot
only once and with a minimum length, and (b) one possible TSP solution (taken from [125]/The Project
Spot).
17.2 Definition of the TSP
24,978 Cities in sweden
Solved in 2004
15,112 Cities in Germany
Solved in 2001
Figure 17.3 Covering cities and towns in Sweden and Germany with near-optimal total length cycles
(taken from [124]/University of Waterloo).
Figure 17.4 A near-optimal tour of 3100 county seats in the USA is 93 466 miles (taken
from [126]/University of Waterloo).
these cities, together with their lengths, are shown in Figure 17.5, where several possible solutions
are presented. The trip that visits Detroit, Chicago, Indianapolis, Columbus and back to Detroit is
the minimum-cost solution in Figure 17.5b. Note that the solution to a TSP may not be unique, and
there may be multiple cycles with the same minimal cost.
Let us now formally define the TSP. Suppose an edge-weighted, undirected graph G is given
as G = (V, E, w), where V is a set of vertices with |V| = n (i.e. V = v1 , v2 , · · · , vn ) and E is a set
of directed edges with |E| = m (i.e. E = e1 , e2 , · · · , em ). A function w maps the edges to their
corresponding weights (e.g. traversal costs, lengths or difficulty); hence w(ei ) = wi , ∀ ei 𝜖 E. In
other words, each edge ei ∈ E is associated with a cost wi . A path in G is defined as a sequence
of edges, where each edge appears only once (all edges may not be included in a path). A cycle
in G is defined as a path that starts and ends at the same node. A Hamiltonian cycle Hc is a
cycle that covers all nodes in G exactly once except for the starting and ending nodes. Based on
these definitions, the goal of the TSP is to find a Hamiltonian cycle with the lowest total weight in
275
276
17 GAs for Route Optimization
a given weighted graph. Formally, we define the TSP as an optimization problem as follows:
minimize
n
∑
wj for ∀ ei ∈ Hc and a given start vertex vs .
(17.1)
j=1
17.3 A Simple Algorithm for the TSP
A straightforward solution for the TSP the compares travelling costs of all possible cycles among
n nodes is shown in Figure 17.6. All possible cycles starting and ending at a selected city, while
visiting the remaining (n − 1) cities, are considered before selecting a cycle with the lowest cost as
the solution.
As can be seen in Figure 17.6, the number of steps needed to find a solution to the TSP
is exponential with respect to the number of cities to be visited and, hence, belongs to the
NP-complete class of problems [128]. The algorithm in Figure 17.6 has a complexity of (n!),
which means that it will need n! iterations to complete for n cities, making it applicable only to
problems with relatively small numbers of cities. In Section 17.5 of this chapter, we will introduce
a computationally lightweight implementation for the TSP class of problems using GAs.
(a)
(b)
(c)
Figure 17.5 TSP solution attempts to find the shortest cycle starting and finishing in Detroit after visiting
Columbus, Indianapolis and Chicago: (a) Detroit, Indianapolis, Chicago, Columbus and Detroit: 990 miles,
(b) Detroit, Chicago, Indianapolis, Columbus and Detroit: 830 miles, (c) Detroit, Chicago, Columbus,
Indianapolis and Detroit: 1060 miles.
Figure 17.6
A simple algorithm to find the lowest-cost Hamiltonian cycles in a graph.
17.5 GA Solution for the TSP
17.4 Variations of the TSP
There are several variations of the TSP, each with a different set of restrictions imposed on its solutions. It is possible that visiting each city exactly once may be too restrictive for certain applications.
For example, if there is a costly edge between two cities (e.g. due to a congested highway, speed
limitations or a traffic condition), it may be preferable to avoid such edges by visiting some of the
remaining cities more than once. In the TSP with multiple visits (TSPM), the aim is to
minimize the total cost of the trip while visiting each vertex at least once (i.e. allowing multiple visits at each city as needed). The path shown in Figure 17.7a is a solution to the TSP which visits
each city exactly once with a cost of 510 miles. If the problem is formulated as a TSPM, the solution
has a total travelled distance of 320 miles with Toledo visited twice, as shown in Figure 17.7b.
In another variation of the TSP, called the clustered TSP (CTSP), the cities are partitioned
into clusters such that a minimum-cost tour of all clusters is to be determined and each city within
a cluster has to be visited before moving to the next cluster. A solution to the CTSP may aim to optimize both an ordering of cities to be visited within each cluster and a sequence in which clusters
are toured [129]. As an example of the CTSP, consider testing integrated circuits where it is common to cluster ports such that any port in a cluster will have the same voltage requirements [123].
To complete the inspection of an integrated circuit, a robotic test probe has to measure the voltages
at all ports of a cluster before moving to the next one. Minimizing the movements of the robot is
equivalent to the CTSP, where every port at every cluster has to be visited with minimal cost.
The TSP variation that divides the graph vertices into subsets and aims to find the minimum-cost
tour by passing through exactly one node from each of the subsets is called the set TSP (STSP). In
some cases the STSP is also referred to as the covering salesmen problem (CSP) since it can
be applied to find a minimum-cost tour among a subset of n cities such that every city not in the tour
is within a predefined distance of a city that is in the tour. In the CSP, each visited city together with
its neighbouring cities constitute a neighbourhood set [130]. As a typical STSP application, consider
the task of testing integrated circuits by measuring several ports, where the ports are divided into
groups such that only one port in each group needs to be measured. Minimizing the movements of
a robotic test probe can be modelled as an STSP.
17.5 GA Solution for the TSP
Genetic algorithms can find solutions for the travelling salesman class of problems, even for cases
where the number of nodes to be covered is in the tens of thousands or more. For simplicity
and without loss of generality, suppose that all cities are interconnected by two-way roads,
(a)
(b)
Figure 17.7 Cycles starting and finishing in Detroit: (a) TSP cycle (all cities are visited exactly
once): Detroit, Toledo, Cleveland and Detroit: 510 miles, (b) TSPM cycle (Toledo is visited more than
once): Detroit, Toledo, Cleveland, Toledo and Detroit: 320 miles.
277
278
17 GAs for Route Optimization
which corresponds to a completely-connected graph G with undirected edges, as presented in
Section 17.2. For this simplified case, a chromosome may encode the cities in the order to be
visited. A fitness function can be defined to represent the total cost of a cycle that is dictated by
a chromosome as its fitness as defined in Eqn. (17.1). A generation of the GA will evolve a set of
individuals in its population, where each individual corresponds to a TSP solution.
In Figure 17.8a, an example graph is shown where the nodes are connected with undirected
edges, each of which is associated with an integer cost value representing the difficulty of traversing the edge. This example uses a completely-connected graph, where every node is connected to every other one with an undirected edge. In this simplified form, the salesman can travel
either from node i to j or from j to i with the same cost. However, in the general case, the TSP
graph does not have to be completely-connected and the edges may not be undirected (i.e. an edge
directed from node i to j does not imply in general that the salesman can also travel from j to i).
The TSP aims to find the minimum cost cycle among all possible cycles starting and ending at
A while covering all edges only once. Even for this simple example, there are 45 possibilities to
consider.
In Figure 17.8b, a connectivity matrix called  is presented, which is derived from the
completely-connected graph in Figure 17.8a. Connectivity matrix  is an (n x n) matrix where
an element [i][j] represents the cost from node i to node j. For example, using a  matrix
of size (5 x 5), the cost associated with the edge from node 0 to 2 (i.e. from node A to node C) is
[0][2] = 12. Since we are using an undirected graph G for this example, an edge from i to j
implies that there is also a connection from j to i with the same cost. As seen in Figure 17.8b,
[0][2] = 12 implies that [2][0] = 12. Diagonal elements of  are zero (i.e. [i][i] = 0
for (i = 0, · · · , n − 1)), since the distance to itself is zero for each city.
[
]
Let us now define a chromosome as the order to visit cities ci , cj , · · · , cx , where we assume that
the last city cx and the first city ci are connected covering all vertices in V. For the example in
Figure 17.8, chromosome length is 5 since there are five cities. The fitness function is the total
weight (i.e. the cost) for the edges selected in a chromosome:
Fitness of a candidate =
m
∑
wi + wR ,
(i=1)
where wi is the weight of the edge in codon cm (i.e. the last one in the chromosome) and wR the
weight of the return edge from the last node to the start. For the example in Figure 17.8, starting
from node A, the TSP is defined as finding the order in which a salesman must visit each node
once and return to A with a minimum total cost. Two candidate solutions are given in Figure 17.9,
namely P1 = A, B, C, D, E, A and P2 = A, B, C, E, D, A. The total cost for cycle P1 is 24, whereas it is
31 for P2 . Between these two candidates, it is clear that P1 is fitter than P2 .
(a)
(b)
Figure 17.8 An example graph, where each undirected edge is associated with an integer cost value
representing the length or difficulty to traverse that edge (a) and its connectivity matrix (b).
17.6 Python Script for Route Optimization Using a GA
Figure 17.9 Two candidate solutions for the
TSP presented in Figure 17.8.
A similar solution applies to the general case of TSP, where some of the nodes are not directly connected to all others and the edges can be undirected, unidirectional or bidirectional. However, an
additional check for the validity of each chromosome will be required since it may inadvertently
include cities that are not directly connected.
17.6 Python Script for Route Optimization Using a GA
In this section we present an implementation of the TSP in a Python script for the simplified version
of the TSP discussed in Section 17.5, where all cities are connected to each other via bi-directional
edges. Suppose we have 15 cities to be visited as shown with their coordinates in Figures 17.10a
and b. The connectivity matrix corresponding to these cities is given in Figure 17.11.
In line 16 of the sample Python script presented below, the input file where city coordinates are stored is identified. A two-dimensional matrix (i.e. a data frame) called df is
(a)
(b)
Figure 17.10 Covering a given set of cities as a TSP example: (a) set of cities used in this example, and
(b) their coordinates.
Figure 17.11
Connectivity matrix for the cities shown in Figure 17.10.
279
280
17 GAs for Route Optimization
created using this input file in line 20. In line 22, a function defined in the gps library called
generate_distance_matrix is invoked to compute the connectivity matrix for the cities.
Chromosome length is defined as the number of cities in line 25. In lines 32 to 50, the chromosome
and population are defined for use later by the GA implementation in the Python library. In lines
53 to 62, the fitness function is implemented as a loop adding the cities’ distances as presented in
the connectivity matrix. The fitness value includes the distance for the return journey from the
last city to the start in line 56. Crossover, mutation and selection operators are defined in lines 69,
72 and 75, respectively. The GA implementation in Python has the following format:
algorithms.eaSimple(pop, toolbox, xoProb, mutProb, gens,
selectCrit)
where pop, toolbox, xoProb, mutProb, gens and selectCrit are the population,
Python library name, crossover probability, mutation probability, number of generations and
selection criteria, respectively. Therefore, the call to invoke the GA in line 84 becomes
algorithms.eaSimple(pop, toolbox, 0.7, 0.2, 40,
halloffame=best_genome)
where halloffame is identified as the best genome to be selected. In lines 103 to 156, the best
solution is plotted on a map.
Note that a regular crossover operator for GA-based TSP implementation may be problematic
if it is not designed carefully. It is possible that a pair of valid chromosomes may generate invalid
offspring after the crossover operator is applied. For example, consider the parent chromosomes
shown in Figure 17.12a defined as [1 2 3 4 5] and [3 2 1 5 4]. Suppose crossover will be applied after
the top two most significant positions. In this case, the offspring will be [1 2 1 5 4] and [3 2 3 4 5],
which are invalid since there are duplicate cities in both cases. A crossover operation where the
codons of an offspring are generated as permutations of the parent codons is implemented in line
69 as
toolbox.register("mate", cxPartiallyMatched)
where partially matched (or mapped) crossover [131] is implemented with two
crossover points as shown in Figure 17.12b. First the middle sections of parents 1 and 2 are
copied into offspring 2 and 1, respectively, which defines a mapping as: 3 ⇔ 2, 4 ⇔ 6, and 5 ⇔ 1.
Then, based on this mapping, the most significant two bits of parent 1 (i.e. 1 and 2) are mapped
into offspring 1 as bits 5 and 3. Similarly the least significant bit of parent 1 (i.e. 6) is mapped into
offspring 1 as bit 4. Offspring 2 is obtained in the same manner.
(a)
(b)
Figure 17.12 (a) Crossover applied to two parents generates invalid offspring with duplicated cities, and
(b) partially matched (mapped) crossover generates valid offspring.
17.6 Python Script for Route Optimization Using a GA
Figure 17.13 (a) Mutation operator generates an invalid offspring
with duplicated cities, and (b) shuffling indices for mutation preserves
the validity of an offspring.
(a)
(b)
A similar concern is also present for the mutation operator. If it is implemented without special
care, a mutated offspring may be invalid. Consider the example in Figure 17.13a, where an offspring
defined as [ 1 2 3 4 5 ] will be mutated at the second most significant position. By replacing this
codon randomly (e.g. from 2 to 4), an offspring may end up as [ 1 4 3 4 5 ], which is invalid since
there is a duplicate codon value. However, implementing mutation as shuffling two codons will
always preserve the validity of an offspring. In Figure 17.13b, mutation operator selects two codons
and shuffles them, resulting in a mutated but valid offspring. This mutation operator that shuffles
the codon values is implemented in line 72 as
toolbox.register("mutate", mutShuffleIndexes, indpb=0.05)
which indicates that the mutation operator is to be executed with a probability of 0.05. In line 87,
the best solution is saved in an array called best_path. In lines 93–101, the cost of the best path
is computed. Between lines 106 and 156, this path is shown on a map using Python libraries.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# The City College of New York, City University of New York
# Written by Ricardo Valdez
# August, 2020
# travelling SALESMAN PROBLEM IMPLEMENTATION USING GA
# THIS IMPLEMENTATION USES deap LIBRARY FUNCTION CALLED eaSimple()
# deap: distributed evolutionary algorithms in python
# ea: evolutionary algorithm
# DEPENDENCIES: deap, basemap, and basemap-data-hires
from deap import creator, base, tools, algorithms
import random
import gps
import pandas as pd
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
cities_file = 'cities.csv'
# load city data into a dataframe df
df = pd.read_csv(cities_file)
## generate a weight matrix where the weights are the distances between cities
distance_matrix = gps.generate_distance_matrix(df)
## set chromosome length to the number of cities
CHROM_LENGTH = len(distance_matrix)
## create a variable called FitnessMin with the create() function.
## FitnessMin inherits from base.Fitness and it will store whether
## the fitness should be minimized or maximized:
##
if weights == -1.0, then minimize the fitness
##
if weights == 1.0, then maximize the fitness
creator.create("FitnessMin", base.Fitness, weights=(-1.0,))
## create a variable called Individual that will determine the
## datastructure used for the chromosome.
## Individual inherits from list and it will store integers.
creator.create("Individual", list, typecode='i', fitness=creator.FitnessMin)
## initialize a new toolbox variable to configure GA parameters:
toolbox = base.Toolbox()
281
282
17 GAs for Route Optimization
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
## configure the population of the GA so that each chromosome is initialized
## as a random permutation of the sequence from 0 to CHROM_LENGTH-1
## range(CHROM_LENGTH): creates a list from 0 to CHROM_LENGTH-1
## range: [0,1,...,9] indices is a random permutation of range
toolbox.register("indices", random.sample, range(CHROM_LENGTH), CHROM_LENGTH)
## assign contents of indices as genes in genome
toolbox.register("genome", tools.initIterate, creator.Individual,
toolbox.indices)
## repeat chromosome assignment for each chromosome in the population
toolbox.register("population", tools.initRepeat, list, toolbox.genome)
## define the fitness function
def TSP_fit_func(chromosome):
## sum up the total distance of the path stored in chromosome
## initialize distance from last element to the first element
distance = distance_matrix[chromosome[-1]][chromosome[0]]
## sum all distances from i to i+1 for all i
for i in range(len(chromosome)-1):
distance += distance_matrix[chromosome[i]][chromosome[i+1]]
## return the path's distance as the fitness
## (at least one comma is needed in return in deap - stupid rule)
return distance,
## configure the evaluate parameter by passing the fitness function
toolbox.register("evaluate", TSP_fit_func)
## implementation of crossover oeprator:
## set the reproduction function to partially matched crossover which
## produces children that are also permutations
toolbox.register("mate", tools.cxPartialyMatched)
## set a 5 percent of mutation, which causes genes in the chromosome to
## be randomly swapped and the result of the mutation is also a permutation
toolbox.register("mutate", tools.mutShuffleIndexes, indpb=0.05)
## set the selection function to tournment selection, which will pick three
## chromosomes at random and select the one with the best fitness
toolbox.register("select", tools.selTournament, tournsize=3)
## set population size to n=100 (n=20 or n=10 see the impact of n)
pop = toolbox.population(n=100)
## have variable best_genome store the chromosome with the best fitness
best_genome = tools.HallOfFame(1)
## run GA to get the solution
#algorithms.eaSimple(pop, toolbox, xo prob, mut prob, gens, store best)
algorithms.eaSimple(pop, toolbox, 0.7, 0.2, 40, halloffame=best_genome)
## store the solution into best_path
best_path = best_genome[0]
## print best_path
print('\nBEST PATH:\n')
for i in range(len(best_path)):
print(df['city'][best_path[i]])
print(df['city'][best_path[0]])
cost = 0
print('\nBEST PATH COST:\n')
for i in range(len(best_path)-1):
tempo = distance_matrix[best_path[i]][best_path[i+1]]
cost+= tempo
print(df['city'][best_path[i]], tempo)
print(df['city'][best_path[0]],distance_matrix[len(best_path)-1][best_path[0]])
cost += distance_matrix[len(best_path)-1][best_path[0]]
print("TOTAL COST: ",cost)
# PLOTTING BEST PATH
# create a figure to draw the plot on
fig = plt.figure(figsize=(10, 10))
## determine the size and position of the map
## resolution is set to low to reduce run time
m = Basemap(projection='lcc', resolution='l',
width=5.5E6, height=3.5E6,
lat_0=39, lon_0=-96,)
17.6 Python Script for Route Optimization Using a GA
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
## set ocean color to blue
m.drawmapboundary(fill_color='#DDEEFF')
## set continent color to orange
m.fillcontinents(color="#FFDDCC")
## draw the coastlines on the map
m.drawcoastlines(color='gray')
## draw country border lines on the map
m.drawcountries(color='black')
## draw state border lines on the map
m.drawstates(color='gray')
## create two empty lists to store the x and y positions for the map plot
X = []
Y = []
for i in range(CHROM_LENGTH):
## convert the (longitude, latitude) pair of the current city to
## (x, y) coordinates for plotting
tempX, tempY = m(df['longitude'][best_path[i]],
df['latitude'][best_path[i]])
## add tempX and tempY to the end of X and Y respectively
X.append(tempX)
Y.append(tempY)
path_color = 'teal'
## draw the shortest path
plt.plot(X,Y,path_color)
## connect the first and last city of the path to close the loop:
## (-1 means the last element of a list)
plt.plot([X[-1],X[0]],[Y[-1],Y[0]],path_color)
## draw points at each city
plt.plot(X,Y,'ok', markersize=7)
for i in range(CHROM_LENGTH):
## label each city on the map
plt.text(X[i],Y[i],' '+df['city'][best_path[i]],fontsize=10,\
fontweight='bold')
## give the plot a title
Plot_title = 'Best Route Found for TSP Using GA - Total cost: ' + str(cost)
plt.title(Plot_title, fontsize=20)
## display the plot
plt.show()
The result for a sample run with the population size set to 10 and using a GA with 10 generations
is shown in Figure 17.14a, where the total cost (i.e. length) of the cycle is 15 507. It is clear that
this is not a good solution since the GA employs a small population and does not have enough
generations to improve its solution. Once the population size and the number of generations are
increased, GA performance is expected to improve significantly. For a population of 100 and using
the GA for 100 generations, the best solution has a cost of 9128, as shown in Figure 17.14b.
(a)
(b)
Figure 17.14 Sample runs of the Python script presented in Section 17.6: (a) with a population of 10 and
running the GA for 10 generations, the solution cost is 15 507, and (b) with population of 100 and running
the GA for 100 generations, the solution cost is 9128.
283
284
17 GAs for Route Optimization
17.7 Matlab Script for Route Optimization Using a GA
A Matlab script implementing a TSP solution using GAs is presented in this section. The input
file for the number of cities are kept the same as in the Python script given in Section 17.6 for
ease of comparison by the reader. In lines 12–17, the input values of the city names with their
latitudes and longitudes are saved in the array called cities. In line 20, the function named
genDistMat is invoked to compute the distance matrix for the input cities. In lines 20 and
25, the functions named TSPlot and TSFitness are defined, respectively. In lines 28 and 29,
options for the GA are defined, where the population is set as the number of cities. In lines 32
to 37, travelling salesman-specific options are defined, where the functions named initPop,
orderedxo and swapMut are used to create the initial population, partially matched (also called
mapped) crossover [131] and mutation by swapping, respectively. These functions are defined
later in the script. In line 40, the GA is invoked by
best_path = ga(FitnessFnc, cities, [ ],[ ],[ ],[ ],[ ],[ ],[ ],
options);
In lines 46–66, the function named get_distance is defined, which computes the distance
between two cities given as a pair of latitudes and longitudes. In lines 68–78, the function named
getDistMat generates a distance matrix for a list of cities. In lines 80–88, the function named
initPop generates a random initial population for the TSP where the chromosomes consist of the
indices to the cities. In lines 90–129, the function named orderexo performs a partially mapped
crossover for two parents. In lines 131–176, the best path is plotted. In lines 178–190, the function
named TSFitness computes the fitness of an individual by computing the total distance between
the cities. In lines 192–208, a mutation operation is performed by the function named swapMut,
by swapping two codons of a chromosome. Finally, between lines 210 and 229, a function named
print_best_path prints the TSP solution on screen.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
% The City College of New York, City University of New York
% Written by Olga Chsherbakova and Hasan Suca Kayman
% August, 2024
% Genetic Algorithms for travelling Salesman
% Function to calculate distance between two geographic coordinates using
% Haversine formula
filename = 'cities.csv';
% Read cities data from CSV file
cities_table = readtable(filename);
% take all columns (Latitude, Longitude)
cities_data = table2array(cities_table(:, 2:3));
% take all columns (City Name, Latitude, Longitude)
cities_data_ = table2cell(cities_table(:, 1:3));
cities = length(cities_data);
% Generate a weight matrix where the weights are the distances
% between cities
distances = generate_distance_matrix(cities_data);
my_plot = @(options,state,flag) travelling_salesman_plot(options, ...
state,flag,cities_data_);
FitnessFcn = @(x) travelling_salesman_fitness(x,distances);
% Set up the options for the genetic algorithm
options = optimoptions(@ga, 'PopulationType', 'custom', ...
'InitialPopulationRange', [1;cities]);
17.7 Matlab Script for Route Optimization Using a GA
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
% Set travelling Salesman Problem specific options
options = optimoptions(options,'CreationFcn',@initialize_population, ...
'CrossoverFcn',@ordered_crossover, ...
'MutationFcn',@swap_mutate, ...
'PlotFcn', my_plot, ...
'MaxGenerations',500,'PopulationSize',60, ...
'UseVectorized',true);
% Run the genetic algorithm
best_path = ga(FitnessFcn,cities,[],[],[],[],[],[],[],options);
% Print the best path
print_best_path(cell2mat(best_path),cities_data_,distances);
% Functions used for TSP GA
% Compute distance between two locations
function distance = get_distance(latitude1,longitude1,latitude2,longitude2)
R = 6373.0; % radius of the Earth in km
% Convert from degrees to radians
lat1 = deg2rad(latitude1);
lon1 = deg2rad(longitude1);
lat2 = deg2rad(latitude2);
lon2 = deg2rad(longitude2);
% Get the change in coordinates
dlon = lon2 - lon1;
dlat = lat2 - lat1;
% calculate the distance between the two coordinates
% using Haversine formula
a = sin(dlat / 2)^2 + cos(lat1) * cos(lat2) * sin(dlon / 2)^2;
c = 2 * atan2(sqrt(a), sqrt(1 - a));
distance = R * c;
% convert distance to miles
distance = distance * 0.621371;
end
% Function to generate distance matrix for a list of cities
function distance_matrix = generate_distance_matrix(cities)
num_cities = size(cities, 1);
distance_matrix = zeros(num_cities, num_cities);
for i = 1:num_cities
for j = 1:num_cities
distance_matrix(i, j) = get_distance(cities(i, 1), ...
cities(i, 2), cities(j, 1), cities(j, 2));
end
end
end
% Initialize population
function pop = initialize_population(NVARS,FitnessFcn,options)
totalPopulationSize = sum(options.PopulationSize);
n = NVARS;
pop = cell(totalPopulationSize,1);
for i = 1:totalPopulationSize
pop{i} = randperm(n);
end
end
% Function to perform ordered crossover to produce a child
% using partially mapped crossover
function childs = ordered_crossover(parents,options,NVARS, ...
FitnessFcn,thisScore,thisPopulation)
nKids = length(parents)/2;
childs = cell(nKids,1);
index = 1;
for i=1:nKids
parent1 = thisPopulation{parents(index)};
parent2 = thisPopulation{parents(index+1)};
index = index + 2;
285
286
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
17 GAs for Route Optimization
child = zeros(1, length(parent1));
start_point = randi(length(parent1));
end_point = mod((start_point + randi(length(parent1) - 2)), ...
length(parent1));
if start_point <= end_point
child(start_point:end_point) = parent1(start_point:end_point);
else
child(start_point:end) = parent1(start_point:end);
child(1:end_point) = parent1(1:end_point);
end
pointer = 1;
for j = 1:length(parent1)
if ∼ismember(parent2(j), child)
while child(pointer) ∼= 0
pointer = pointer + 1;
end
child(pointer) = parent2(j);
end
end
childs{i} = child;
end
end
% Function to visualize the best path on a map using geoplot
function state = travelling_salesman_plot(options,state,flag,cities_data_)
[score,i] = min(state.Score);
best_path = state.Population{i};
persistent geoAx plotHandle;
% Extract latitudes and longitudes for the best path
lats = cell2mat(cities_data_(best_path, 2));
longs = cell2mat(cities_data_(best_path, 3));
% Close the loop by adding the starting city to the end
lats = [lats; lats(1)];
longs = [longs; longs(1)];
% Create the GeographicAxes only once
if isempty(geoAx) || ∼isvalid(geoAx) || ...
∼isvalid(ancestor(geoAx, 'figure'))
clf;
%figure('Position', [100, 100, 800, 600]);
geoAx = geoaxes;
geobasemap(geoAx, 'landcover');
plotHandle = [];
end
% Delete the previous plot if it exists
if ∼isempty(plotHandle) && isvalid(plotHandle)
delete(plotHandle);
end
% Plot the new path and store the plot handle
plotHandle = geoplot(geoAx, lats, longs, '-o', 'Color', ...
'red', 'LineWidth', 2);
% Update the title with the current best cost
title(geoAx, ['Best Route Found for TSP using GA - Total Cost: ',...
num2str(score,'%.2f')]);
% Add city names to the map
city_names = cities_data_(best_path, 1);
for i = 1:length(best_path)
text(geoAx, lats(i), longs(i), city_names{i}, ...
'VerticalAlignment', 'bottom', 'HorizontalAlignment', ...
'center', 'FontSize', 8, 'FontWeight', 'bold');
end
end
17.8 Concluding Remarks
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
% Function to evaluate the total distance of the path represented by
% the chromosome
function scores = travelling_salesman_fitness(x,distances)
scores = zeros(size(x,1),1);
for j = 1:size(x,1)
city = x{j};
total_distance = distances(city(end),city(1));
for i = 2:length(city)
total_distance = total_distance + distances(city(i-1),city(i));
end
scores(j) = total_distance;
end
end
% Function to perform mutation by swapping two genes in the chromosome
function mutationChildren = swap_mutate(parents ,options,NVARS, ...
FitnessFcn, state, thisScore,thisPopulation,mutationRate)
mutationChildren = cell(length(parents),1);
for i=1:length(parents)
parent = thisPopulation{parents(i)};
p = ceil(length(parent) * rand(1,2));
while p(1)==p(2)
p = ceil(length(parent) * rand(1,2));
end
child = parent;
child(p(1)) = parent(p(2));
child(p(2)) = parent(p(1));
mutationChildren{i} = child;
end
end
% Function to print the best path with distances
function print_best_path(best_path, cities_data, distance_matrix)
num_cities = length(best_path);
best_path = circshift(best_path,num_cities - find(best_path==2) + 1);
fprintf('BEST PATH COST:\n');
fprintf('%-15s %-15s\n', 'City', 'Distance');
total_distance = 0;
for i = 1:num_cities
current_city = best_path(i);
next_city = best_path(mod(i, num_cities) + 1);
current_city_info = cities_data(current_city, :);
next_city_info = cities_data(next_city, :); % ASK
distance = distance_matrix(current_city, next_city);
total_distance = total_distance + distance;
fprintf('%-15s %-15.2f\n', ...
current_city_info{1}, distance);
end
fprintf('Total Distance: %.2f\n', total_distance);
end
Two sample runs of the Matlab script presented in this section is displayed in Figures 17.15a
to d. Running the Matlab script with a population of 60 and for 600 generations, the total cost of
the solution is 8 435.07 miles, whereas the solution for a population of 10 and 10 generations is
12 760.53 miles. Since for the second run the population is too small and the GA is not run for
enough generations, it performs much worse than the first run, as expected.
17.8 Concluding Remarks
The TSP represents a class of optimization problems where the main goal is to find the
minimum-cost cycle covering a set of points connected to each other via edges with differing cost
values. Many complex optimization problems which require planning, scheduling, logistics and
287
288
17 GAs for Route Optimization
(a)
(b)
(c)
(d)
Figure 17.15 Sample runs of the Matlab script presented in Section 17.7: the orders of cities and map
displays. (a) and (b) The population size is 60 for 600 generations of GA; (c) and (d) the population size is
10 for 10 generations of GA.
packing are suitable candidates to be formulated as TSPs. An algorithmic solution for the TSP is
computationally infeasible for large graphs since the computational cost grows factorially with
the number of nodes.
In this chapter we present a GA-based solution to the TSP such that the order of the cities to be
visited constitutes a chromosome, and the length of the cycle is the fitness of a solution. We present
simplistic Python and Matlab scripts implementing this GA-based approach to the TSP. The computational complexity of this GA-based solution is linear with the number of cities, although an
absolute minimum is not guaranteed for large problems (with tens of thousands of nodes) due to
the probabilistic nature of GAs. However, a near-minimum solution is almost always found.
17.9 Exercises for Chapter 17
(a)
(b)
(c)
(d)
Figure 17.16 (a) Caribbean nations, (b) former British Commonwealth countries, (c) West Pacific nations,
and (d) cities for Appleton smart phones.
17.9 Exercises for Chapter 17
1) Generate a cycle that visits all Caribbean countries listed in Figure 17.16a. Note that the cycle
may start from any country but must visit every country only once and return to the starting
point.
2) The King of England has to visit the former British Commonwealth countries. Find an itinerary
so his trip is minimum length and only visits each country once. Use Figure 17.16b for
coordinates.
3) Maersk Ocean Freight Services is planning to deliver cars and electronics in the West Pacific.
How can they achieve this stopping in each country exactly once? Coordinates of the countries
are given in Figure 17.16c.
4) Appleton wants to train employees on the usage of their newest smart phones. What is the best
possible route? The coordinates are in Figure 17.16d.
289
291
18
Evolutionary Methods
18.1 Introduction
In this chapter, we outline prominent evolutionary computation methods, including particle
swarm optimization in Section 18.2, differential evolution in Section 18.3 and
grammatical evolution in Section 18.4. For each one, we present a numerical example to
illustrate the mechanics of implementing the method in any language and software environment.
We would like to point out the ever powerful parallelizable characteristics of evolutionary
algorithms, which make them excellent candidates for very-large-scale problems. One can easily
envision that, in the near future, millions of wetware computers running a particle swarm
optimization algorithm in parallel can solve a space travel problem with low overhead. Similarly,
the best flight control program for an autonomous spacecraft exploring galaxies can be generated
by a grammatical evolution algorithm using a realistic grammar and running in parallel on a few
million biological computers.
18.2 Particle Swarm Optimization
Based on mechanisms observed in biological organizations, swarm artificial intelligence (SAI) is
used to control and analyse the collective behaviour of decentralized autonomous systems [132].
The agents in SAI systems interact with their nearest neighbours and their external environment
to manifest a global intelligence.
Self-organization can be defined as a capacity that emerges from interactions among lower-level
components that have not been explicitly designed to achieve a goal [133]. For example, social
wasps are able to build an elaborate nest with each worker depositing building materials based
on a local environment. Typically, wasps are more likely to add to a corner area of an existing
wall than start building a new wall [134]. With the deposit of new material, the nest configuration is altered, which then provides new information for other wasps that visit that location later.
This indirect communication through the environment is called stigmergy. Consequently, even
simple insects, such as wasps, are able to build very complex structures. Self-organized systems
may rely on positive and negative feedback, amplification of random fluctuations and multiple
direct or indirect stigmergic interactions among individuals to obtain a dynamic system exhibiting
emergent properties with bifurcation (i.e. capable of obtaining a new stable solution if and when
the properties of a system change).
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
292
18 Evolutionary Methods
Particle swarm optimization (PSO) is an SAI optimization technique inspired by social and
collective behaviour observed in nature, such as bird flocking and fish schooling. It was developed
as a tool for solving problems in which the best solution can be represented as a point in the space
[105]. PSO is a robust stochastic optimization technique based on the movement and intelligence
of swarms. PSO implements social interaction principles to solve multi-dimensional optimization
problems. In PSO, if one of the individuals finds a good solution, the remaining individuals make
use of that knowledge.
PSO is a multi-agent parallel search method where each particle flying over a search space adjusts
its flight path based on its own experience (i.e. historical data) as well as the experiences of other
particles. At a given time t, a particle j, with a position Pjt and velocity V(t), keeps track of the best
fitness it has achieved so far individually (called Pjbest ) and the best value global value obtained
g+1
g
so far by the swarm overall (called G best ). Let Pj be the new position for particle j and Pj be its
current position. We evolve the speed for particle j as
g
g
Vg+1 = w Vg + C1 R1 (Pjbest − Pj ) + C2 R2 (G best − Pj ),
(18.1)
where Vg+1 and Vg are the new and current speeds for particle j, respectively, and w is a real number in [0, 1] and is a weight factor for inertia. Real numbers C1 and C2 are in [0, 4], and the real
numbers R1 and R2 are in [0, 1].
In Eq. (18.1), the momentum of a particle j is controlled by w as a trade-off between a particle’s
previous experience and the overall swarm experience. For w = 0, particle j does not consider any of
its previous knowledge, whereas w = 1 implies that particles do not easily change their directions.
In velocity updates, R1 influences Pjbest , while R2 impacts G best . For contribution of experience to the
search, C1 represents self-confidence of a particle and C2 indicates the particle’s confidence in the
g+1
g
motion of the entire swarm. The new position of j is defined as Pj = Pj + Vg+1 . The personal and
global best fitness values are updated if the new position of particle j has a better fitness. Regardless
of its fitness, particle j moves to the new location.
Variations of PSO
There are several different types of PSO algorithms that have been developed in order to address
a diverse range of engineering applications. One commonly use swarm optimization method,
called discrete PSO (DPSO), solves combinatorial optimization problems over a discrete-valued
search space [135]. In this algorithm, the position and velocity of each particle are denoted by
discrete values that could be real numbers or integers. Another version of the PSO algorithm,
called neighbourhood search PSO (NSPSO), considers both local neighbours and the global
swarm to find its trial solutions. In NSPSO, a particle first explores the local neighbourhood of
each of its neighbours before considering the global solution space [136]. In a different variation,
called neighbourhood guaranteed convergence PSO (GCPSO), the structure of social networks is
implemented by preserving the diversity within the swarm where a distance limit is applied when
finding the best next positions [137].
18.2.1 Applications of PSO
PSO can be easily adopted to solve a diverse range of engineering problems. For example, it has
been popular for solving maintenance and path discovery types of applications in communication
and computer networks, especially for power constrained mobile nodes. PSO can minimize the
communication cost by obtaining optimal or near-optimal transmission paths while considering
18.2 Particle Swarm Optimization
energy consumption and traffic intensity [138]. PSO can be useful in power management problems
in large-scale wireless ad-hoc networks, where nodes are clustered into smaller subnetworks with
only relatively few links between them. PSO finds stable clusters of nodes to ensure energy-efficient
and stable communication in the network [139].
In data mining applications, large amounts of information are handled such that chunks of
data with similar characteristics need to be grouped together. PSO can automatically determine
an optimal number of clusters needed for accurate classification of input data. Each particle represents a data element located at the centre of a cluster, where distance is measured by the similarity
between two data elements. A corresponding fitness function is used to promote clusters that are
denser and with larger numbers of data points [140].
18.2.2 PSO Operation
PSO uses a number of agents (i.e. particles) that constitute a swarm flying over a search space
looking for the best solution:
A particle adjusts its flying based on its own flying experience as well as the flying experience of
other particles.
● Each particle keeps track of its coordinates in the solution space which are associated with
the best solution (fitness) that has been achieved so far by that particle. This value is called the
personal best: Pj(best) .
● Another important value tracked by the PSO is the best value obtained so far by any particle in
the swarm. This value is called the global best: G best .
●
g
At generation g, a particle j has a position Pj and a velocity of v(t). Each particle knows its own
personal best position and the global best position of all particles. In PSO, evolving the speed of a
g
g+1
particle j from generation g to (g + 1) (i.e. Pj → Pj ) is accomplished as follows:
g
g
Vg+1 = w ⋅ Vg + C1 ⋅ R1 ⋅ (Pjbest − Pj ) + C2 ⋅ R2 ⋅ (G best − Pj ),
(18.2)
where the following PSO optimization parameters are used:
Vg+1 : new speed for the particle
● V : current speed for the particle
g
g
● P : current position of the particle
j
● w: weight factor for inertia (a real number in the range [0, 1])
(best)
● R : influence of P
in velocity update (a random real number in the range [0, 1])
1
j
best in velocity update (a random real number in the range [0, 1]) [105]
● R : influence of G
2
● C : self-confidence of a particle in velocity update (a real number in the range [0, 4])
1
● C2 : swarm-confidence of the entire population in velocity update (a real number in the range
[0, 4])
(best)
● P
: the personal best position that particle j has flown over so far
j
best : the global best position that any particle in the swarm has flown over so far
● G
●
Parameter w controls the momentum of a particle:
w << 1: very little momentum is kept from the last generation.
w = 0: no knowledge from last time-step is kept; each generation is independent.
● w = 1: particles do not easily change their directions – typically used to explore larger areas in
the beginning.
●
●
293
294
18 Evolutionary Methods
w can be decreased as a function of number of generations (e.g. more exploration at the beginning
and more trust on swarm intelligence towards the end).
● Typically inertia weight w is reduced as the number of generations grow (e.g. from 0.9 to 0.4)
[105].
●
C1 is the self-confidence of a given particle: how much of a particle’s own experience contributes
to the search at a given generation (self-exploration). C2 represents swarm confidence, the motion
of all particles in the swarm at a given generation (swarm-exploration). Typically, C1 and C2 are set
to 1.5.
Computing a New Position for a Particle
For the two-dimensional case, let a new position for a particle j be:
g+1
g
g+1
g
Pj,x = Pj,x + Vg+1,x
Pj,y = Pj,y + Vg+1,y ,
where
g+1
Pj,x : new position for particle j in generation (g + 1) in x dimension
g
● P : current position for particle j in generation g in x dimension
j,x
● V
g+1,x : new speed for particle j in x dimension
g+1
● P
: new position for particle j in generation (g + 1) in y dimension
j,y
g
● P : current position for particle j in generation g in y dimension
j,y
● Vg+1,y : new speed for particle j in y dimension
●
After finding the new position for the particle j, PSO calculates the fitness for this new position.
If the new fitness is better than the personal and/or global best fitness, it updates those parameters accordingly. Regardless of the fitness value, particle j moves to the new location in the new
generation. A simple implementation of the PSO algorithm is given in Figure 18.1.
Figure 18.1
A simple implementation of PSO.
18.2 Particle Swarm Optimization
Benchmark for PSO: Rosenbrock Function
A benchmark function for PSO is the Rosenbrock function [141], which is defined for the
two-dimensional case as
f (x, y) = (1 − x2 ) + 100( y − x2 )2 .
The goal for PSO is to find the minimum value for the Rosenbrock function, which is in (0, 0),
as shown in Figure 18.2. Let us now demonstrate the steps of PSO operation for this benchmark.
We use a simple PSO with five particles and the parameters shown in Figure 18.3a. Let the initial
random positions of the particles be as given in Figure 18.3b. In this example, we focus on particle
3 and observe how it evolves as the PSO algorithm progresses. The initial position of particle 3 is
selected randomly as P30 = (0.49, −1.92) (Figure 18.3b). The fitness for particle 3 is
f (P30 ) = (1 − 0.49)2 + 100(−1.92 − (0.49)2 )2 = 3143.43.
Figure 18.2
Rosenbrock function as a benchmark for PSO.
Figure 18.3 Numerical example for PSO using
Rosenbrock function: (a) PSO parameter values,
and (b) randomly selected initial positions of
five particles.
(a)
(b)
295
296
18 Evolutionary Methods
pbest
The personal best for particle 3 at (0.49, −1.92) is f (P3 ) = 3143.43. The global best for the swarm
pbest
pbest
will be selected as the minimum among f (P1 ), · · · , f (P5 ). Let us now evolve particle 3 from
generation 0 to 1. For an inertia weight of w = 0.9, R1 = 0.8659 and R2 = 0.7489, the new speed for
particle 3 is computed as follows:
Vx = 0.9 ⋅ 0 + 1.5 ⋅ 0.8659 ⋅ (0.49 − 0.49) + 1.5 ⋅ 0.7489 ⋅ (1.87 − 0.49)
= 1.55 mod 1 = 0.55
Vy = 0.9 ⋅ 0 + 1.5 ⋅ 0.8659 ⋅ (−1.92 + 1.92) + 1.5 ⋅ 0.7489 ⋅ (1.49 + 1.92)
= 3.83 mod 1 = 0.83.
Using the new speed, we now compute the new position for particle 3 as follows:
P31 (x) = 0.49 + Vx = 1.04
P31 ( y) = −1.92 + Vy = −1.08.
The fitness for the new position P31 = (0.49, 1.04) is computed as f (P31 ) = 504.63. Since
f (P31 ) < f (P3(best) ), the new position is better and will be stored as the new personal best. Therefore, for
particle 3 we have the new location P31 = (0.49, 1.04) with a speed of V = (0.55, 0.83) and a personal
best of P3(best) = (1.04, −1.08). Fitness values, personal and global best positions for five particles are
shown in Figure 18.4. We see that the global best is position G best = (0.12, 1.44), which is the position of particle 1 with a fitness value of f (G best ) = 40.92. New positions of all five particles are shown
in Figures 18.5 (note that the computations for particles other than #3 are not shown for brevity). As
part of the PSO algorithm, all particles move to their new positions regardless of the fitness values.
Let us now evolve particle 3 from generation 1 to generation 2. At generation 1, we have:
P31 = (1.04, −1.08), P3(best) = (1.04, −1.08), G best = (0.35, 1.08) and f (G best ) = 40.92. For an inertia
weight of w = 0.73, R1 = 0.43 and R2 = 0.9925, the new speed for particle 3 is
Vx = 0.73 ⋅ 0.55 + 1.5 ⋅ 0.43 ⋅ (1.04 − 1.04) + 1.5 ⋅ 0.9925 ⋅ (0.12 − 1.04)
= 0.96
Vy = 0.73 ⋅ 0.83 + 1.5 ⋅ 0.43 ∗ (−1.08 + 1.08) + 1.5 ⋅ 0.9925 ⋅ (1.44 + 1.08)
= 4.35 mod 1 = 0.35.
Figure 18.4
The fitness values and personal and global best positions for all particles.
18.2 Particle Swarm Optimization
Figure 18.5 New positions of the five particles (note that, for brevity, the calculations for the other
particles are not shown).
Using the new speed, we now compute the new position for particle 3 as follows:
P32 (x) = 1.04 + Vx = 0.08
P32 ( y) = −1.08 + Vy = −0.73.
The fitness for the new position P32 = (0.08, −0.73) is computed as f (P32 ) = 160.34. Since
f (P32 ) < f (P3(best) ), the new position is better than current personal best position and will be stored
as such. Therefore, for particle 3 we have a new location of P32 = (0.08, −0.73), with a speed of
V = (−0.96, 0.35) and a personal best of P3(best) = (0.08, −0.73). Fitness values, personal and global
best values and the new positions of all five particles are shown in Figures 18.6 and 18.7. We see
that the new global best is position G best = (0.35, 1.08), which is the new position of particle 1 with
a fitness value of f (G best ) = 1.17.
After running the PSO process for either a pre-determined number of generations or until a
user-defined condition is met (e.g. verifying that the PSO is not making significant improvements
to global fitness values for new generations), the PSO concludes by choosing the last G best as its
final solution.
Figure 18.6
The fitness values and personal and global best positions for all particles.
297
298
18 Evolutionary Methods
Figure 18.7
New positions of the five particles.
18.2.3 Remarks for PSO
PSO is a robust stochastic optimization technique based on the movement and intelligence of
swarms. It was developed in 1995 by James Kennedy (social psychologist) and Russell Eberhart
(electrical engineer) [105]. PSO applies the concept of social interaction to multi-dimensional
optimization problems. In PSO, if one of the individuals finds a good solution, all the other
individuals also use that knowledge. Implementation of PSO is relatively simple with only a few
parameters to adjust (e.g. the particle velocity). PSO is a multi-agent parallel search method,
where each particle (i.e. agent) flies over the search space. The population of particles is called
a swarm. PSO is easily parallelizable and therefore ideal for problems with large search spaces.
PSO algorithms are a natural fit in controlling the topology of autonomous drones, where the
drone positions can be modelled as a swarm of particles [142]. Due to its special way of moving
the particles between generations, it may avoid being stuck at a local extreme points compared to
traditional AI methods.
18.3 Differential Evolution
Differential evolution (DE) is an evolutionary computation technique originally proposed by Price
and Storn in 1995 to solve high-order Chebychev polynomial equations [106]. DE is different from
other evolutionary computation methods such as GA and PSO in that it does not use classical
crossover and mutation operators but generates offspring by a new differential operator.
This new operator first generates a difference vector of two randomly chosen individuals
in the population to create a new individual. Because a different difference vector is created for
every individual, DE has a better chance of generating diverse populations compared to traditional
evolutionary computation methods, where typically a fixed probability distribution determines perturbations. DE is self-organizing and requires very little input from the user.
In DE, a population with N individuals operating in a d-dimensional space, an individual i
is represented as a d-dimensional vector. At a given time t, DE creates two solutions for each
individual i, called variant and trial solutions. If the trial solution performs better
g
than individual i, it replaces individual i at time (t + 1). Let us represent i at generation g as Pi .
g
g+1
DE evolves this individual to the next generation of (g + 1) (i.e. Pi → Pi ) as follows. First a
18.3 Differential Evolution
variant solution Vi is created by picking three other individuals from the population (say with
the indices of k, r and m). These three indices must be distinct and different than i. A variant for
individual i at generation g is calculated as follows [143]:
g
g
g
Vi = Pm + E ⋅ (Pk + Pr ),
where E is a scaling factor to control the amount of amplification of the difference vector (typically E ∈ [0, 2]). After variant Vi is created, DE generates a trial solution called Ui as follows. For
dimension x, real numbers radnx and rnbr (both random and in the range of [0, 1]) and a crossover
constant (called CR), a real number in the range of [0, 1] are utilized. If randx < CR or rnbr = 0,
variation solution Vi for dimension x is assigned as the x dimension of the trial solution Ui . Otherg
wise, the existing value of Pix (i.e. the x dimension of the existing solution) is used in determining
the trial solution Ui in dimension x. The other dimensions of the trial solution are computed in a
g
g
similar manner. For Ui , if fitness f (Ui ) is better than f (Pi ), it replaces individual Pi in generation
g
(g + 1); otherwise Pi is kept as the solution for the next generation.
18.3.1 Different Versions of DE
Different versions of DE algorithms are proposed by researchers, each aiming to improve its
outcome accuracy and computational performance when applied to a particular problem.
The main differences among these algorithms stem from the methods used in selection of individuals for a variant solution, the number of pairs of individuals to be selected and the methods used
to combine the individuals. For example, in a DE variation called DE/rand/Np/bin, Np pairs of
individuals are selected randomly for a binomial combining method. In a binomial combination,
one parent is chosen with probability pa from a vector and the other with probability (1 − pa ) [144].
Other variations of DE use the best individual in the current population as the base vector (e.g. in
DE/best/Np/bin) or calculate parents using at least two difference vectors, one obtained from the
best vector and the parent vector while others are calculated using randomly selected vectors (e.g.
in DE/current-to-best/Np/bin). It is also possible to combine the trial solution with an individual by
using a multi-point combination operator that randomly selects many cut points at which vectors
should be combined to obtain offspring [145].
18.3.2 Applications of DE
Because of its simple computational requirements, DE has a wide range of applications that require
multi-objective optimization, where the goal is to find an optimal (or near-optimal) solution that
satisfies multiple, and often contradicting, goals at the same time. Due to its parallelizable nature,
DE can be performed such that each candidate solution in a population is evaluated in parallel.
For problems with large search spaces, computation of a fitness function may require a long time
(e.g. days of computation for each individual). This is typical, for example, in space exploration
applications, where each step involves millions of iterations of heavy computation for each of the
thousands of individuals in a population for an experiment that requires millions of steps [142, 146].
18.3.3 A Simple Algorithm for DE
A simple algorithm to implement DE is shown in Figure 18.8. First, a random population of N
individuals is created as candidate solutions. Then, each individual is evaluated to obtain its fitness
value. At each generation of DE, a variant and a trial solution are created for each individual in the
299
300
18 Evolutionary Methods
Figure 18.8 A simple algorithm implementing DE (here smaller fitness values are assumed to be
preferable in selecting individuals for the next generation).
population. DE decides if it will replace an individual i at step (t + 1) by its trial solution using their
fitness values. The algorithm continuously evolves a population of individuals for a predetermined
time or until a satisfactory solution of the fitness function is reached.
Let us consider a two-dimensional solution space whose candidate solutions can be evaluated
by a fitness function f (extension to a higher-dimensional space is straightforward). After the
g
g
g+1
initialization of the population, individual Pi evolves from generation g to (g + 1) (i.e. Pi → Pi )
by creating a variant solution Vi = (vix , viy ):
Select three distinct individuals from the current population as k, r and m such that all three
indices and i are distinct: i ≠ k ≠ r ≠ m.
g
g
g
● Individuals P , Pr and Pm will be used to calculate a variant as follows:
k
●
g
g
g
vi = Pm + E ⋅ (Pk − Pr ),
g
g
g
where Pm is called the base vector, (Pk − Pr ) is called the difference vector and E (typically
E ∈ [0, 2]) is the scaling factor controlling amplification of the difference vector.
● For a two-dimensional variant, we have:
g
g
g
vix = Pmx + E ⋅ (Pkx − Prx )
g
g
g
viy = Pmy + E ⋅ (Pky − Pry ).
g
g
g
For the special case of E = 1, the variant becomes Vi = Pm + Pk − Pr . Using the variant solution
Vi , create a trial solution Ui = (uix , uiy ) as follows:
Generate randx and randy – random real numbers in the range (0, 1).
Generate rnbr - a random integer in the range (0, 1) (for the general case, rnbr is a random integer
in the range (0, d − 1)).
● The CR (crossover constant) parameter is selected as a real number in the range (0,1).
●
●
18.3 Differential Evolution
A trial solution for the x dimension is determined as follows:
if (randx ≤ CR OR rnbr = 0): variant solution for x dimension is used as the trial solution’s x
dimension: select the trial solution as uix = vix .
● if (randx > CR AND rnbr ≠ 0): trial solution’s x dimension is the same as the x dimension of the
g
individual from generation g: uix = Pix .
●
Similarly, a trial solution for the y dimension is obtained as follows:
if (randy ≤ CR OR rnbr = 1): variant solution for y dimension is used as the trial solution’s y
dimension: select the trial solution as uiy = viy .
● if (rand > CR AND rnbr ≠ 1): trial solution’s y dimension is the same as the y dimension of the
y
g
individual from generation g: uiy = Piy .
●
Figure 18.9 shows examples of generating a trial solution for all four possible cases (i.e. only x,
only y or both x and y dimension solutions being selected from the trial solution).
After constructing a trial solution Ui , its fitness f (Ui ) is calculated. Then f (Ui ) is compared with
g
the fitness of the current individual f (Pi ). Then a selection is performed as follows. If the trial
solution has better fitness than the current individual from generation g, then it is selected for
generation (g + 1); otherwise, the current individual from generation g is selected such that:
g
g+1
if (f (Ui ) < f (Pi )) then Pi
g+1
g
else
Pi = Pi .
Figure 18.9
= Ui
Examples of generating a trial solution – all possible cases.
301
302
18 Evolutionary Methods
18.3.4 Numerical Example: Maximum of sinc by DE
Let us use DE to find the maximum of the sinc function [147], which is defined as
√
sin (x2 + y2 )
.
√
(x2 + y2 )
It is well known that traditional methods (e.g. Newtonian approaches or gradient ascent and
its variations) are likely get stuck at a local maximum. The value of the sinc function is analytic
everywhere, as the removable singularity at zero has the limit value of one (Figure 18.10a).
In practice, sinc functions with more than 50 dimensions are common. In this simplified example,
a two-dimensional sinc function will be considered in the range [0, 10]. The extension of this
example to a sinc function with a high-dimensional space is straightforward. In a typical realistic
case, the population size for DE is selected as 10 times the number of dimensions. For our simple
example, the initial population consists of a randomly selected four points. The parameters used
in the example are presented in Figure 18.10b.
The initial randomly chosen population of DE and their depiction as vectors in the sinc function are shown in Figures 18.11a and b, respectively. In Figure 18.11a, Pj0 represents position j in
generation 0 as a potential maximum value for the sinc function.
(a)
(b)
Figure 18.10 (a) Benchmark for DE: finding maximum of sinc function and (b) Parameters of DE
application to sinc function.
(a)
Figure 18.11
function.
(b)
(a) Initial population of DE and (b) the vectors depicting the initial population in the sinc
18.3 Differential Evolution
Evolving Position 0 from Generation 0 to 1: Let us evolve position 0 from generation 0 to 1
(i.e. P00 → P01 ). A variant V = (vx , vy ) is constructed as follows:
●
Suppose we pick: m = 1, k = 2, r = 3; rnbr = 0; randx = 0.8; randy = 0.6. Recall that CR = 0.7.
We have:
vx = 0 + 0.8 ⋅ (3 − 1) = 1.6
vy = 4 + 0.8 ⋅ (7 − 2) = 8.
For the x dimension of U = (ux , uy ), since (randx ≤ CR) is false but (rnbr = 0), we have:
ux = vx = 1.6. For the y dimension, since (randy ≤ CR), uy = vy = 8.
● Let us now calculate the fitness for the trial solution and compare it with the fitness for the
position from generation 0: f (5, 9) = −0.0743 and f (1.6, 8) = 0.1169, which means that for a maximization problem f (U) > f (P00 ). In other words, the position (1.6, 8) suggested by the trial solution
for generation 1 is a better position than P00 = (5, 9). Therefore, we use the position given by the
trial solution U for generation 1: P01 = (1.6, 8).
●
Evolving Position 1 from Generation 0 to 1: Evolving position 1 from generation 0 to 1 (i.e.
P10 → P11 ), V = (vx , vy ) is constructed as follows:
●
Suppose we pick: m = 2, k = 0, r = 3; rnbr = 1; randx = 0.4; randy = 0.8. Recall that CR = 0.7.
vx = 3 + 0.8 ⋅ (5 − 1) = 6.2
vy = 7 + 0.8 ⋅ (9 − 2) = 12.6 = 10 (limited by Rangey ).
For the x dimension of U = (ux , uy ), (randx ≤ CR) is true, and we have: ux = vx = 6.2. Since
(randy ≤ CR) is false but rnbr = 1, we have: uy = vy = 1.
0
● f (0, 4) = −0.1892 and f (6.2, 10) = −0.06, which means that f (U) > f (P ). In other words, the posi1
tion (6.2, 10) suggested by the trial solution for generation 1 is a better position than P10 = (0, 4).
Therefore, we use the position given by the trial solution U for generation 1: P11 = (6.2, 10).
●
Evolving Position 2 from Generation 0 to 1: Evolving position 2 from generation 0 to 1 (i.e.
P20 → P21 ), V = (vx , vy ) is constructed as follows:
●
Suppose we pick: m = 3, k = 0, r = 1; rnbr = 0; randx = 0.4; randy = 0.9. Recall that CR = 0.7.
vx = 1 + 0.8 ⋅ (5 − 0) = 5
vy = 2 + 0.8 ⋅ (9 − 4) = 6.
For the x dimension of U = (ux , uy ), (randx ≤ CR) is true, then: ux = vx = 5. For the y dimension,
1
= 7.
since (randy ≤ CR) is false and rnbr = 0, we have: uy = P2y
0
● f (5, 7) = −0.0852 and f (3, 7) = 0.1276, which means that f (U) < f (P ). In other words, the posi2
tion (5, 7) suggested by the trial solution for generation 1 is not a better position than P20 = (3, 7).
Therefore, we use position P20 for generation 1: P21 = (3, 7).
●
Evolving Position 3 from Generation 0 to 1: Evolving position 3 from generation 0 to 1 (i.e.
P30 → P31 ), variant V = (vx , vy ) is constructed as follows:
●
Suppose we pick: m = 0, k = 1, r = 2; rnbr = 0; randx = 0.5; randy = 0.9. Using CR = 0.7, we
have:
vx = 5 + 0.8 ⋅ (0 − 3) = 2.6
●
vy = 9 + 0.8 ⋅ (4 − 7) = 6.6.
Let us construct a trial solution U = (ux , uy ). For the x dimension, since (randx ≤ CR) is true,
we have: ux = vx = 2.6. For the y dimension, since (randy ≤ CR) is false and rnbr = 1, we have:
uy = vy = 6.6.
303
304
18 Evolutionary Methods
●
f (2.6, 6.6) = 0.1021 and f (1, 2) = 0.35, which means that f (U) < f (P30 ). In other words, the position (2.6, 6.6) suggested by the trial solution for generation 1 is not a better position than P30 =
(1, 2). Therefore, we use position P30 for generation 1: P31 = (1, 2).
Candidate positions for generations 0 and 1 are shown in Figures 18.12 and 18.13.
Evolving Position 0 from Generation 1 to 2: Now, let us evolve position 0 from generation 1 to
2 (i.e. P01 → P02 ). Let us first construct a variant V = (vx , vy ) as follows. Suppose we pick: m = 1, k = 3,
r = 2; rnbr = 1; randx = 0.6; randy = 0.9. With CR = 0.7, we have:
vx = 6.2 + 0.8 ⋅ (1 − 3) = 4.6
vy = 10 + 0.8 ⋅ (2 − 7) = 6.
Let us construct a trial solution U = (ux , uy ). For the x dimension, since (randx ≤ CR) is true but
1
(rnbr = 0), we have: ux = P0x
= 4.6. For the y dimension, since (randy ≤ CR) is false but rnbr = 1,
we have: uy = vy = 6.
● The fitness for the trial solution: f (4.6, 6) = −0.1266 and f (1.6, 8) = 0.1169, which means the position (4.6, 6) suggested by the trial solution for generation 2 is a better position than P01 = (1.6, 8).
Therefore, we use the position given by trial solution U for generation 2: P02 = (4.6, 6).
●
Evolving Position 1 from Generation 1 to 2: Evolving position 1 from generation 1 to 2 (i.e.
P11 → P12 ), V = (vx , vy ) is constructed as follows:
Figure 18.12 Candidate positions at generation 1 (Pj0 is a candidate position j in generation 0 as the initial
random population and Pj1 is a position in generation 1).
(a)
(b)
Figure 18.13 Positions of candidates for maximizing the sinc function: (a) initial positions and
(b) positions at generation 1.
18.3 Differential Evolution
●
Suppose we pick: m = 3, k = 0, r = 2; rnbr = 1; randx = 0.4; randy = 0.8. With CR = 0.7, we
have:
vx = 1 + 0.8 ⋅ (1.6 − 3) = −0.12 → 0 (has to be between 0 and Rangex )
vy = 2 + 0.8 ⋅ (8 − 7) = 2.8.
Let us construct a trial solution U = (ux , uy ). For the x dimension, since (randx ≤ CR) is true,
we have: ux = vx = 0. For the y dimension, since (randy ≤ CR) is false but rnbr = 1, we have:
uy = vy = 2.8.
1
● f (0, 4) = −0.1892 and f (6.2, 10) = −0.06, which means that f (U) > f (P ). In other words,
1
the position (0, 2.8) suggested by the trial solution for generation 2 is a better position than
P11 = (6.2, 10). Therefore, we use the position given by trial solution U for generation 2:
P12 = (0, 2.8).
●
Evolving Position 2 from Generation 1 to 2: Evolving position 2 from generation 1 to 2 (i.e.
P21 → P22 ), V = (vx , vy ) is constructed as follows:
●
Suppose we pick: m = 1, k = 3, r = 0; rnbr = 0; randx = 0.4; randy = 0.8. Recall that CR = 0.7.
vx = 6.2 + 0.8 ⋅ (1 − 1.6) = 5.72
vy = 10 + 0.8 ⋅ (2 − 8) = 5.2.
Let us construct a trial solution U = (ux , uy ). For the x dimension, since (randx ≤ CR) is true,
we have: ux = vx = 5.72. For the y dimension, since (randy ≤ CR) is false and rnbr = 0, we have:
2
uy = P1y
= 7.
1
● f (5.72, 7) = −0.0415 and f (3, 7) = 0.1276, which means that f (U) < f (P ). In other words, the
2
position (5, 7) suggested by the trial solution for generation 2 is not a better position than
P21 = (3, 7). Therefore, we use position P21 for generation 2: P22 = (3, 7).
●
Evolving Position 3 from Generation 1 to 2: Evolving position 3 from generation 1 to 2 (i.e.
P31 → P32 ), V = (vx , vy ) is constructed as follows:
●
Suppose we pick: m = 2, k = 0, r = 1; rnbr = 1; = 0.4; randy = 0.2. Recall that CR = 0.7:
vx = 3 + 0.8 ⋅ (1.6 − 6.2) = −0.68 → 0
vy = 7 + 0.8 ⋅ (8 − 10) = 5.4.
For the x dimension of U = (ux , uy ), since (randx ≤ CR) is true, we have: ux = vx = 0. For the y
dimension, since (randy ≤ CR) is true, we have: uy = vy = 5.4.
1
● f (0, 5.4) = 0.1431 and f (1, 2) = 0.3518, which means that f (U) < f (P ). In other words, the posi3
tion (0, 5.4) suggested by the trial solution for generation 2 is not a better position than P31 = (1, 2).
Therefore, we use position P31 for generation 1: P32 = (1, 2).
●
Positions at generations 1 and 2 are shown in Figures 18.14 and 18.15. More generations are
processed in a similar manner, and the best solution is selected at the last generation as the
solution.
18.3.5 Remarks for DE
DE is one of the most recent AI techniques introduced into the field. It does not use the traditional
genetic operators such as crossover and mutation; instead it uses variant solutions to decide on the
next generation’s search direction. Because of this feature, DE has an improved chance of generating diverse populations since it uses different perturbations for each individual. The algorithms
implementing DE are computationally light-weight, which are shown to be effective in financial
305
306
18 Evolutionary Methods
Figure 18.14 Candidate positions at generation 2 (Pj1 is a candidate position j in generation 1 and Pj2 in
generation 2).
(a)
(b)
Figure 18.15 Positions of candidates for maximizing the sinc function: (a) at generation 1 and (b) at
generation 2.
applications. Similar to other evolutionary computation methods, DE also requires a well-defined
fitness function that matches the needs of the application domain.
18.4 Grammatical Evolution
Grammatical evolution (GE) is a powerful technique to generate solutions to problems where the
solutions are computer programs. As for any programming language, GE requires that a syntax
must be defined for acceptable solutions, specific for each application. The programs generated by
GE can be implemented using any standard programming language. GE is a powerful AI tool, especially for problems where a large number of parameters, each with its own search space, need to
be considered before reaching a solution. An example GE application would be designing an effective autonomous driving protocol for self-driven vehicles, where hundreds of input variables with
different parameter values are required to be evaluated in order to make safe decisions in real time.
18.4.1 A Simple Algorithm for GE
Figure 18.16 shows a simple algorithm for implementing GE. In this algorithm, first an initial
population of randomly selected candidate solutions is created. This step is common in most
18.4 Grammatical Evolution
Figure 18.16
Rosenbrock function as a benchmark for PSO.
evolution-based computation techniques such as GA, PSO and DE. A candidate solution is
represented by a chromosome, which consists of an array of codons. A program is generated
using the codon values of the individual and the language rules defined for this application. In GE,
Backus-Naur form (BNF) [148, 149] is used to define a set of rules for describing the grammar
for the application at hand. Derivation of a program for a given chromosome based on the BNF
rules is described in the next section.
Once a program is derived from a chromosome, its fitness value must be calculated. Since the
solution is a program, there has to be a medium where this program will execute and generate
results for the application. Similar to the other evolutionary methods, based on the performance
of the candidate, a fitness value will be assigned to it. For example, if the candidate program is
intended for an algorithmic trading application, it will be executed over historical financial data.
The fitness could be the profit that a candidate program generates over a period of time. The implementation of GE in Figure 18.16 uses standard selection, crossover and mutation operations seen
in traditional GAs to promote fitter individuals from one generation to the next. Evolutionary operations are repeated until a satisfactory solution is found or a stopping criterion is reached.
18.4.2 Definition of GE
Since the solutions in GE are computer programs, a grammar for valid syntax defining acceptable
programs is needed for each specific application. In BNF the syntax of a programming language is
given by a tuple {N, T, P, S}, where N is the set of non-terminal variables, T is the terminals
(e.g. literal symbols such as numbers or values of Boolean variables), P is a set of production
rules and S ∈ N is the start symbol [148]. Every rule in P uses a meta-symbol ∶∶= to define
how to expand the non-terminals. In a production rule, the symbol “|” separates alternative definitions for non-terminals. A pair of symbols < and > define the start and the end of a non-terminal,
respectively. Elements of N are mapped to T using a sequence of rules. In these rules, an element
of N may appear either on the left or right side of a rule. However, terminals cannot be on the left
307
308
18 Evolutionary Methods
side of a rule. As an example, consider the following production rule with two choices:
< int >∶∶= < digit > ∣ < int >< digit >
where int and digit are two non-terminals. In this example, int is defined to have two choices: either
a non-terminal digit or itself (i.e. int) followed by digit. Let us now consider another example of a
rule with 10 choices:
< digit >∶∶= 0 ∣ 1 ∣ 2 ∣ 3 ∣ 4 ∣ 5 ∣ 6 ∣ 7 ∣ 8 ∣ 9
where digit is defined as one of the terminals 0 to 9. These two rules make up the BNF grammar
rule for representing any integer [150].
Choices in a production rule can be numbered. In a chromosome, codon values can be mapped
on the choices in rules by using a modulo operator as follows:
choice = Cn modulo Rn
where Cn is the value of a codon of a chromosome, and Rn is the number of choices for this
non-terminal. Suppose int = S is the starting symbol, and the chromosome is given as
1001
1111
0100
1110
1001
whose decimal representation is
9
15
4
14
9
For the first codon we have (9 modulo 2 = 1) since rule 0 has 2 choices. The start symbol int = S is
expanded by using rule 0 and choice 1 as
< int >< digit >
We continue by applying the leftmost derivation to expand the non-terminals. Similarly, in order to extend < int > we use (15 modulo 2 = 1), which dictates that the leftmost
< int > is expanded with rule 0 choice 1 yielding < int >< digit >< digit >. Now the new
leftmost < int > is expanded using rule 0 choice 0 to yield
< digit >< digit >< digit >
Now the leftmost non-terminal is a digit with 10 possible choices. The next codon value to consider
is 14. Since (14 modulo 10 = 4), at this point, the solution is
4 < digit >< digit >
After applying 9 to next digit, we have the solution as
48 < digit >
At this point, we finish traversing our string of codons, but our solution still has non-terminals
to process. We continue by reusing the codons from the beginning, hence again applying rules
encoded by codon to obtain the final integer of 488. Wrapping a sequence for codon reuse is consistent with gene-overlapping in biological processes.
Variations of GE
Some GE implementations use PSO to explore the search space of possible solutions in an attempt
to improve the fitnesses of population members when moving from one generation to the next.
18.4 Grammatical Evolution
This variation of GE is called a grammatical swarm [151]. In 𝜋Grammatical Evolution
(𝜋GE), a different order for derivation of candidate solutions from BNF is proposed [152]. In
𝜋GE, codon values are used to specify which non-terminal will be developed next, instead of the
traditional order of processing from left to right. This variation provides a more flexible means to
generate candidate solutions and, hence, is expected to be more effective for applications where
population diversity is critical. To improve the search power of GE, weighted hierarchical GE
imposes a hierarchy based on the genotype and encodes grammar symbols with a varying number
of bits based on the relative expressive power of those symbols [153].
Applications of GE
GE has been applied to solving a broad range of engineering problems in a wide range of disciplines.
For example, GE can be used in swarm robotics to manage multiple robots operating as a single
system. The collective behaviour of autonomous robots can be synthesized by evolving a set of
rules for each individual robot. A program that dictates actions for a robot may be derived from
BNF grammar rules [154].
In finance, algorithmic trading policies can be represented by BNF grammars to populate
different meta-parameters in a variety of technical indicators [155]. Similarly, bankruptcy,
corporate analysis and credit classification are within the scope of GE applications [156]. In bioinformatics, GE can be employed to recognize eukaryotic promotors in identification of biological
genes [157].
In telecommunications, GE can generate programs that are capable of recognizing malicious
activities on a network and setting appropriate access policies to prevent unauthorized use of network resources [158]. A BNF grammar can be designed to consider network traffic in a way that
allows for generated programs to detect cyberattacks [159]. GE may also be used to formulate security rules that balance potentially conflicting goals (e.g. security risks, exposure of network nodes
and convenient access to network resources for authorized users). In this application, the BNF
grammar of GE can infer multi-level security policies to determine trustworthy entities and their
appropriate need-to-know access rights [160].
BNF Notation
BNF notation expresses a grammar of a language as production rules as follows [149]:
terminals: items that appear in a language (e.g. +,−,0,1,..,9, etc.)
non-terminals: they are terms that can be expanded into one or more terminals and
non-terminals
● A grammar can be represented by a tuple G = {N, T, P, S}, where
– N: a set of non-terminals; for example, N = {exp, op, cond}
– T: a set of terminals; for example, T = {1, 2, ..., 9, (, ), if , else}
– P: a set of production rules; for example,
< exp >∶∶=< exp > | < int >< cond >< int >
– S: a start symbol (S ∈ N); for example, S =< exp >
●
●
Santa Fe Trail Problem
The Santa Fe trail problem is an instance of the artificial ant problem [107], where the objective
is to find a computer program to control an artificial ant so that the ant can find all food pieces.
Figure 18.17 shows the Santa Fe trail, which contains 89 pieces of food placed on a trail over a
32 × 32 toroidal grid. The trail has 144 squares and 21 turns. An example BNF for the Santa Fe trail
309
310
18 Evolutionary Methods
Figure 18.17 Santa Fe trail: there are 89 pieces of food on a 32x32
toroidal grid.
Figure 18.18
An example BNF for the Santa Fe trail [107].
Figure 18.19
[107].
Best-performing output program generated using the BNF grammar given in Figure 18.8
problem is shown Figure 18.18. An artificial ant can perform one of the following actions in one
time unit:
turn left, turn right, move ahead
An artificial ant can look into the square it is currently facing and decide if the square contains a
food piece or is empty. If there is food in the square the ant moves into, it is assumed that the ant
automatically consumes the food, without requiring a specific command to do so. The best reported
solution completes the trail in 545 steps [161]. The best performing program generated using the
the BNF grammar shown in Figure 18.18 is shown in Figure 18.19 [107].
A Simple BNF for an Artificial Ant
Let us now consider an example BNF as shown in Figure 18.20, which is simpler than the one
shown in Figure 18.18. Figure 18.21 presents an example chromosome (i.e. a genotype) and the
expression tree (i.e. a phenotype) generated using this chromosome and the BNF grammar of
Figure 18.20. Step-by-step generation of this expression tree is presented in Figures 18.22 to 18.23.
Derivation starts by considering the start symbol S expression of the BNF, which is S = <expr>
for this example (Figure 18.22). After <expr> is located at the root of the expression tree, we
observe that there are two choices for expanding the non-terminal called <expr>, namely if
18.4 Grammatical Evolution
Figure 18.20
A simple BNF for an artificial ant.
Figure 18.21 The genotype and the corresponding expression tree generated using the BNF grammar of
Figure 18.20.
food ahead then <act> else <act> and <expr>. To resolve this choice, we use the
first codon of the genome, which is a 0, which implies that if food ahead then <act>
else <act> will be placed below <expr>. At this point, terminals if food ahead then
will be placed below <expr>, but the non-terminal <act> must be resolved. The non-terminal
<act> has three choices, namely <move><expr>, <move> and <move><act>. This choice
is resolved by the next codon of the genome, which is a 2. Therefore, the third choice of
<move><act> will be placed into the expression tree below <act>. Derivation continues
similarly until all non-terminals in the tree are resolved in terms of terminals using the codon
values from the genome.
The example above is constructed in such a way that the codon values of the genotype shown
in Figure 18.20 were defined in advance to match the number of choices available for each BNF
rule. However, in general, the codon values of a genotype may be arbitrarily large. For these
cases, we can use the modulo function to obtain proper codon values as dictated by the number of
choices in a given BNF rule that is being considered. For example, Figure 18.24 shows a genotype
which has arbitrarily large integer values. In this case, the modulo operator will be applied to
resolve the choices of the non-terminal to be evaluated. For example, after <expr> is located at
the root of the expression tree, the two choices for expanding the non-terminal called <expr>,
namely if food ahead then <act> else <act> and <expr>, are resolved by using
the codon value of 0, which is if food ahead then <act> else <act>. Terminals
if food ahead then will be placed below <expr>, but the non-terminal <act> must be
resolved. The non-terminal <act> has three choices, namely <move><expr>, <move> and
<move><act>. This choice is resolved by the next codon of the genome, which is a 5. This
means that the selection will be 5 modulo 3 = 2, which is <move><act>. In general, when
there are c choices for a non-terminal and a codon value of r, the choice will be (r modulo
c = 2). For simplicity, genotype values in Figure 18.24 are selected such that they generate the
same program in Figure 18.23n by applying the BNF of Figure 18.20.
311
312
18 Evolutionary Methods
Figure 18.22
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Steps (a)–(h): Generating a phenotype from a given genotype using a given BNF grammar.
18.4 Grammatical Evolution
Figure 18.23
program.
(i)
( j)
(k)
(l)
(m)
(n)
Steps (i)–(m): Phenotype generation continued from Figure 18.22, and (n): the resulting final
Note that due to recursion, it may be that a phenotype does not complete. In this case the
conversion is aborted, similarly to cases in biology.
Figure 18.25 shows the steps for computing the fitness of the program in Figure 18.23n over a
simple trail with three food pieces. In this example, fitness of a candidate program may be defined
as the number of items consumed by an artificial ant whose movements are controlled by the
candidate program.
313
314
18 Evolutionary Methods
Figure 18.24 An example chromosome with arbitrarily large codon values which generates the same
program as the chromosome in Figure 18.22 (using the modulo function to generate BNF rule selections).
Figure 18.25 Computing the fitness of the program given in the top-left corner using a simple trail with
three food pieces.
18.4.3 A Simple GA to Implement GE
Let us now understand how a phenotype, which is a valid program generated by a given BNF, will
be used as part of an evolutionary algorithm. For simplicity, let us first suppose that we will use
GAs as the method to evolve the genotypes.
As we studied in Chapter 14, in implementing a GA, an initial population of candidate solutions
is first formed, each of which possesses chromosomes with randomly chosen codon values. The
fitness of each candidate solution is then evaluated, and the population is sorted based on the candidate fitness values (Figure 14.3). According to a pre-determined criterion, a set of candidates is
then selected as parents to generate the offspring for the next generation by employing crossover
and mutation operators. The fitness of the offspring is computed, and the previous population and
the offspring are temporarily merged; however, only the best-performing part of the population is
kept for future generations. Based on their fitness, it is possible that the non-performing offspring
will not be included in the new populations.
18.4 Grammatical Evolution
In adaptation of GE into the GA framework, each phenotype is evaluated by running it in the
environment that is native to the given BNF grammar. For example, if the BNF was written for a
C++ program, the phenotype will be a C++ program which will be a compiled and executed, and
the output will be evaluated for a given criterion. If, on the other hand, the BNF was written for a
Python script, then it will be directly executed in the environment specified for the BNF. Suppose
that the BNF was written to perform algorithmic trading as described in Chapter 4. The phenotype
will then be placed into an electronic trading environment and its performance will be observed.
Based on a desired criterion (e.g. the amount of profit it makes), a fitness value will be assigned to
the phenotype (i.e. the candidate solution in the GA population).
18.4.4 Remarks on GE
GE is a powerful technique for spanning large solution spaces. It is applicable to problems where
the solution is a computer program. Since GE is independent of programming languages and program execution environments, it is a ubiquitous technique applicable to a wide range of applications, from the financial arena to space exploration missions. The depth of BNF grammar and the
computational cost of the fitness function calculation for each genome define the computational
complexity of the GE.
315
317
Appendix A
ANNs with Bias
A.1
Introduction
In ANN models representing real-life problems, a bias input is needed to adjust the weights with
more accuracy and faster convergence to reduce output errors. For inclusion of a bias as part of
an ANN model, one has to add an additional input at each hidden layer and the output layer
of the ANN. Similar to regular ANN synapses connecting neurons, the weights of bias synapses
are also associated with weights that are initially assigned random values. Bias inputs, therefore,
become intermixed with link weights through summation and activation functions in forward and
back propagation calculations. During training, bias weights are updated based on error values in a
manner similar to synapse weights. In this appendix, we will derive the equations used in updating
weights of neuron synapses and bias links in an ANN employing input biases at every iteration of
training.
A.2
Training with Bias Input
A hidden-layer neuron with a bias input is given in Figure A.1, where a bias link with weight of B is
added as one of the neuron inputs. Let us now formulate the weights for the incoming and outgoing
links of a neuron that includes bias inputs. As shown in Figure A.2, the summation function works
as follows:
e1 = (x1 ⋅ w1 ) + (x2 ⋅ w4 ) + b1
e2 = (x1 ⋅ w2 ) + (x2 ⋅ w5 ) + b2
e3 = (x1 ⋅ w3 ) + (x2 ⋅ w6 ) + b3 .
An activation function f (.) is applied to the summed signals at each hidden neuron as follows:
a1 = f (e1 )
a2 = f (e2 )
a3 = f (e3 ),
where the a1 , a2 and a3 values are determined based on the activation function employed at the
neurons, as presented in Eqs. (2.1) and (2.3) in Chapter 2. Similarly, at the output layer, the output
neuron has three inputs which are aggregated as:
e4 = (a1 ⋅ w7 ) + (a2 ⋅ w8 ) + (a3 ⋅ w9 ) + b4 .
After e4 is obtained, it is fed to an activation as a4 = f (e4 ).
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
318
Appendix A ANNs with Bias
Figure A.1
A neuron with bias input at the output layer of an ANN.
Figure A.2
Summation and activation in hidden-layer neurons.
A.3
Forward Propagation
A simple ANN with two inputs, one hidden layer and one output layer utilizing bias inputs is
presented in Figure A.3. Similar to the derivation presented in Section 2.3.4 in Chapter 2, we have
x1 , x2 and bias as the inputs to the neuron.
Figure A.3
Example ANN with two inputs with bias, one hidden layer and one output with bias.
A.3 Forward Propagation
A.3.1
Forward Propagation from Input to Hidden Layer
Let us now consider the forward propagation step in the ANN training process. For the derivations in this section, as in Section 2.5, we will use a simple ANN with one hidden layer as given in
Figure A.3. For forward propagation from input to hidden layer of this simple ANN, we have:
(2)
(1)
(1)
= x1 ⋅ w(1)
Z(11)
11 + x2 ⋅ w21 + b1
(2)
Z(12)
= x1 ⋅ w(1)
+ x2 ⋅ w(1)
+ b(1)
12
22
2
(2)
Z(13)
= x1 ⋅ w(1)
+ x2 ⋅ w(1)
+ b(1)
13
23
3
Using the definition of sigmoid activation given in Eq. (2.1) in Chapter 2, we have the following
outputs of the hidden-layer neurons:
(2)
(1)
(1)
= x1 ⋅ w(1)
Z(11)
11 + x2 ⋅ w21 + b1
(2)
Z(12)
= x1 ⋅ w(1)
+ x2 ⋅ w(1)
+ b(1)
12
22
2
(2)
Z(13)
= x1 ⋅ w(1)
+ x2 ⋅ w(1)
+ b(1)
13
23
3
At the output-layer neuron, aggregation of the three input synapses and the bias yields
(2)
(2)
(2)
(2)
(2)
(2)
Z (3) = a(2)
11 ⋅ w1 + a12 ⋅ w2 + a13 ⋅ w3 + b1
Finally at the output layer, using a sigmoid activation function generates ŷ as
1
ŷ = f (Z (3) ) =
1 + e−Z
1
=
.
−(a(2)
⋅w(2)
+a(2)
⋅w(2)
+a(2)
⋅w(2)
+b(2)
11
1
1 )
12
2
13
3
1+e
A.3.2
Neuron Back Propagation with Bias Input
Let us go over the steps of back propagation over a neuron with a bias input for a neuron with
two inputs xa and xb and a bias incoming over synapses using weights wa , wb and B, respectively,
generating an output of ŷ , as shown in Figure A.4. The output of a neuron and its partial derivation
are defined as
𝜕 ŷ
= f ′ (e).
ŷ = f (e),
𝛿e
For e = xa wa + xb wb + B, partial derivatives of e with respect to wa , wb and B are as follows:
𝜕e
= xa ,
𝜕wa
𝜕e
= xb ,
𝜕wb
𝜕e
= 1.
𝜕B
For error J defined as J = 12 (y − ŷ )2 , the partial derivative of J with respect to wa is
𝜕 ŷ
𝜕 ŷ 𝜕e
𝜕J
= −(y − ŷ )
= −(y − ŷ )
.
𝜕Wi
𝜕Wi
𝜕e 𝜕Wi
Let us define 𝛿 = −(y − ŷ ) f ′ (e) and plug it into the above error equation:
𝜕e
𝜕J
=𝛿
= 𝛿xa .
𝜕wa
𝜕wa
The partial derivative of J with respect to wb is
𝜕 ŷ
𝜕 ŷ 𝜕e
𝜕J
= −(y − ŷ )
= −(y − ŷ )
.
𝜕wb
𝜕wb
𝜕e 𝜕wb
319
320
Appendix A ANNs with Bias
Figure A.4 A neuron with two inputs xa
and xb and a bias incoming over synapses
using weights wa , wb and B, respectively,
generating an output of ŷ .
For 𝛿 = −(y − ŷ ) f ′ (e), we have
𝜕e
𝜕J
=𝛿
= 𝛿xb .
𝜕wb
𝜕wb
Similarly, the partial derivative of J with respect to B is
𝜕 ŷ
𝜕 ŷ 𝜕e
𝜕J
= −(y − ŷ ) Z = −(y − ŷ )
.
𝜕B
𝜕B
𝜕e 𝜕B
Again, using 𝛿 = −(y − ŷ ) f ′ (e), we have
𝜕e
𝜕J
=𝛿
= 𝛿.
𝜕B
𝜕B
Using these values, the weights will be updated for a learning rate of 𝛼 as follows:
wa = wa − 𝛼
𝜕J
𝜕J
𝜕J
, wb = wb − 𝛼
, B=B−𝛼 .
𝜕wa
𝜕wb
𝜕B
321
Appendix B
Sleep Study ANN with Bias
B.1 Inclusion of Bias Term in ANN
Including the bias input into a simple ANN, we have the ANN shown in Figure B.1.
A neuron with bias for back propagation is shown in Figure B.2.
1
J = (y − ŷ )2
2
The partial derivative of J with respect to b is
𝜕J
𝜕b
̂
̂
𝜕y
𝜕e
= −(y − ŷ ) 𝜕b
= −(y − ŷ ) 𝜕𝜕ey 𝜕b
.
For 𝛿 = −(y − ŷ )f ′ (e), we have
𝜕J
𝜕b
B.1.1
= 𝛿 ⋅ 1 = 𝛿.
Inclusion of Bias in Matrices
Suppose there are k different bias values, one for each neuron at layer i with k neurons:
]
[
b(i) = b1 b2 · · · bk .
For n samples, the dimensions of the bias matrix become the number of samples by the number of
neurons at layer i:
⎡ b1 b2 · · · bk ⎤
⎢
⎥
b b · · · bk ⎥
.
b(i) = ⎢ 1 2
⎢· · ·
⎥
⎢b b · · · b ⎥
k⎦
⎣ 1 2
B(i) has n rows, each with k elements. At each row a, the jth element is the same, ending up with
each column of B(i) having the same values of bias as shown above. Similarly, for the 𝛿 values, we
have the following matrix:
⎡ 𝛿 1 𝛿 2 · · · 𝛿k ⎤
⎢
⎥
𝛿 𝛿 · · · 𝛿k ⎥
.
𝛿 (i) = ⎢ 1 2
⎢· · ·
⎥
⎢𝛿 𝛿 · · · 𝛿 ⎥
k⎦
⎣ 1 2
For the mean(𝛿 (i) ), a matrix can be defined as:
[
]
mean(𝛿 (i) ) = mean(𝛿 (1) ) mean(𝛿 (2) ) · · · mean(𝛿 (k) ) .
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
322
Appendix B Sleep Study ANN with Bias
Figure B.1
A simple ANN with bias input for sleep-study example.
Figure B.2
A neuron with bias during back propagation for sleep-study example.
B.1.2
Forward Propagation with Biases
Recall the input and output values for the normalized case:
⎡ 0.3 1 ⎤
X = ⎢ 0.5 0.2 ⎥
⎢
⎥
⎣ 1 0.4 ⎦
⎡0.75⎤
Y = ⎢0.82⎥ .
⎢
⎥
⎣ 093 ⎦
Randomly selected W (1) and W (2) matrices are
W (1) =
[
]
−0.2 0.8 0.3
0.6 −0.4 −0.2
⎡−0.1⎤
W (2) = ⎢ 0.2 ⎥ .
⎢
⎥
⎣ 0.1 ⎦
Randomly selected b(1) and b(2) matrices are
[
]
[ ]
b(1) = 0.2 −0.5 0.4 b(2) = 0.1
] ⎡0.2 −0.5 0.4⎤
⎡0.3 1 ⎤ [
−0.2 0.8 0.3
+ ⎢0.2 −0.5 0.4⎥
Z (2) = XW (1) + B(1) = ⎢0.5 0.2⎥
⎢
⎥ 0.6 −0.4 −0.2
⎢
⎥
⎣ 1 0.4⎦
⎣0.2 −0.5 0.4⎦
⎡0.7 −0.6 0.3⎤
= ⎢0.2 −0.2 0.5⎥ .
⎢
⎥
⎣0.2 0.1 0.6⎦
Using a rectilinear activation function:
⎡0.7 0 0.3⎤
a(2) = f (Z (2) ) = ⎢0.2 0 0.5⎥
⎢
⎥
⎣0.2 0.1 0.6⎦
B.1 Inclusion of Bias Term in ANN
⎡0.7 0 0.3⎤ ⎡−0.1⎤ ⎡0.1⎤ ⎡0.06⎤
Z (3) = a(2) W (2) + B(2) = ⎢0.2 0 0.5⎥ ⎢ 0.2 ⎥ + ⎢0.1⎥ = ⎢0.13⎥
⎢
⎥⎢
⎥ ⎢ ⎥ ⎢
⎥
⎣0.2 0.1 0.6⎦ ⎣ 0.1 ⎦ ⎣0.1⎦ ⎣0.17⎦
⎡0.06⎤
ŷ = f (Z (3) ) = ⎢0.13⎥
⎢
⎥
⎣0.17⎦
∑
J = 12 (y − ŷ )2 = 12 ((0.75 − 0.06)2 + (0.82 − 0.13)2 + (0.93 − 0.17)2 ) = 1.53
⎡0.69⎤ ⎡1⎤ ⎡−0.69⎤
𝛿 (3) = −(y − ŷ ). ∗ f (Z (3) ) = − ⎢0.69⎥ . ∗ ⎢1⎥ = ⎢−0.69⎥ ,
⎢
⎥ ⎢ ⎥ ⎢
⎥
⎣0.76⎦ ⎣1⎦ ⎣−0.76⎦
where . ∗ represents element-wise multiplication.
𝛿
(2)
=𝛿 W
(3)
(2)T
1 0 1⎤
⎡−0.69⎤ [
] ⎡
⎢
⎥
⎢
. ∗ f (Z ) = −0.69 −0.1 0.2 0.1 . ∗ 1 0 1⎥
⎢
⎥
⎢
⎥
⎣−0.76⎦
⎣ 1 1 1⎦
(2)
⎡0.07 0 −0.07⎤
= ⎢0.07 0 −0.07⎥
⎢
⎥
⎣0.08 −0.15 −0.08⎦
⎡0.74 0.24 0 ⎤ ⎡−0.69⎤ ⎡−0.84⎤
𝜕J
(2)T (3)
⎢ 0
=
a
𝛿
=
0 0.14⎥ ⎢−0.69⎥ = ⎢−0.11⎥
⎥
⎥ ⎢
⎢
⎥⎢
𝜕W (2)
⎣0.29 0.51 0.62⎦ ⎣−0.76⎦ ⎣−1.02⎦
[
] ⎡0.07 0 −0.07⎤ [
]
0.3 0.5 1 ⎢
0.14 −0.15 −0.14
𝜕J
T (2)
⎥
=X 𝛿 =
0.07 0 −0.07 =
⎥
1 0.2 0.4 ⎢
0.12 −0.06 −0.12
𝜕W (1)
⎣0.08 −0.15 −0.08⎦
[
]
𝜕J
= mean 𝛿 (3) = −0.71
(2)
𝜕b
[
]
0.07 −0.05 −0.07
𝜕J
(2)
=
mean
𝛿
=
.
1
0.2
0.4
𝜕b(1)
Let us now compute the new weights for the normalized values with bias using a learning rate of
𝛼 = 0.25:
[
]
[
]
−0.2 0.8 0.3
0.14 −0.15 −0.14
𝜕J
=
−
0.25
W (1) = W (1) − 𝛼
0.6 −0.4 −0.2
0.12 −0.06 −0.12
𝜕W (1)
[
]
−0.24 0.84 0.34
=
0.57 −0.39 −0.17
W (2) = W (2) − 𝛼
b(1) = b(1) − 𝛼
[
]
⎡−0.84⎤ ⎡0.11⎤
−0.1 0.2
𝜕J
⎢−0.11⎥ = ⎢0.23⎥
=
−
0.25
⎢
⎥ ⎢
⎥
0.1
𝜕W (2)
⎣−1.02⎦ ⎣0.36⎦
[
]
[
]
𝜕J
= 0.2 −0.5 0.4 − 0.25 0.07 −0.05 −0.07
𝜕b(1) [
]
= 0.18 −0.49 0.42
[ ]
[
] [
]
b(2) = b(2) − 𝛼 𝜕b𝜕J(2) = 0.1 − 0.25 −0.71 = 0.28 .
323
324
Appendix B Sleep Study ANN with Bias
Normalized – forward propagation with bias (epoch 2):
] ⎡0.18 −0.49 0.42⎤
⎡0.3 1 ⎤ [
−0.24 0.84 0.34
+ ⎢0.18 −0.49 0.42⎥
Z (2) = XW (1) + B(1) = ⎢0.5 0.2⎥
⎢
⎥ 0.57 −0.39 −0.17
⎢
⎥
⎣ 1 0.4⎦
⎣0.18 −0.49 0.42⎦
⎡0.68 −0.63 0.35⎤
= ⎢0.17 −0.15 0.56⎥ .
⎢
⎥
⎣0.17 0.19 0.69⎦
Using a rectilinear activation function:
⎡0.68 0 0.35⎤
a(2) = f (Z (2) ) = ⎢0.17 0 0.56⎥
⎢
⎥
⎣0.17 0.19 0.69⎦
⎡0.68 0 0.35⎤ ⎡0.11⎤ ⎡0.28⎤ ⎡0.48⎤
Z (3) = a(2) W (2) + B(2) = ⎢0.17 0 0.56⎥ ⎢0.23⎥ + ⎢0.28⎥ = ⎢ 0.5 ⎥
⎢
⎥⎢
⎥ ⎢
⎥ ⎢
⎥
⎣0.17 0.19 0.69⎦ ⎣0.36⎦ ⎣0.28⎦ ⎣0.59⎦
⎡0.48⎤
ŷ = f (Z (3) ) = ⎢ 0.5 ⎥
⎢
⎥
⎣0.59⎦
∑
J = 12 (y − ŷ )2 = 12 ((0.75 − 0.48)2 + (0.82 − 0.5)2 + (0.93 − 0.59)2 ) = 0.291.
Note that the error in epoch 1 was 1.53, which is significantly reduced in epoch 2.
Normalized – back propagation with bias (epoch 2)
⎡0.75 − 0.48⎤ ⎡1⎤ ⎡−0.27⎤
𝛿 (3) = −(y − ŷ ). ∗ f (Z (3) ) = − ⎢ 0.82 − 0.5 ⎥ . ∗ ⎢1⎥ = ⎢−0.32⎥ ,
⎥
⎢
⎥ ⎢ ⎥ ⎢
⎣0.93 − 0.59⎦ ⎣1⎦ ⎣−0.34⎦
where . ∗ represents element-wise multiplication.
1 0 1⎤
⎡−0.27⎤ [
] ⎡
𝛿 (2) = 𝛿 (3) W (2)T . ∗ f (Z (2) ) = ⎢−0.32⎥ −0.11 0.23 0.36 . ∗ ⎢1 0 1⎥
⎢
⎥
⎢
⎥
⎣−0.34⎦
⎣1 1 1⎦
⎡−0.03 0 −0.11⎤
= ⎢−0.04 0 −0.12⎥
⎢
⎥
⎣−0.04 −0.8 −0.12⎦
⎡0.68 0.17 0.17⎤ ⎡−0.27⎤ ⎡ −0.3 ⎤
𝜕J
(2)T (3)
=a 𝛿 =⎢ 0
0 0.19⎥ ⎢−0.32⎥ = ⎢−0.06⎥
⎢
⎥⎢
⎥ ⎢
⎥
𝜕W (2)
⎣0.35 0.56 0.69⎦ ⎣−0.34⎦ ⎣−0.51⎦
[
] ⎡−0.03 0 −0.11⎤
0.3 0.5 1 ⎢
𝜕J
T (2)
=X 𝛿 =
−0.04 0 −0.12⎥
⎥
1 0.2 0.4 ⎢
𝜕W (1)
⎣−0.04 −0.8 −0.12⎦
[
]
−0.07 −0.08 −0.21
=
−0.05 −0.03 −0.17
B.1 Inclusion of Bias Term in ANN
[
]
𝜕J
= mean 𝛿 (3) = −0.31
(2)
𝜕b
[
]
−0.04 −0.03 −0.11
𝜕J
(2)
=
mean
𝛿
=
.
𝜕b(1)
For epoch 2, we can now compute the new weights and biases for the normalized values using a
learning rate of 𝛼 = 0.25:
[
]
[
]
−0.24 0.84 0.34
0.07 −0.08 −0.21
𝜕J
W (1) = W (1) − 𝛼
=
−
0.25
0.57 −0.39 −0.17
0.05 −0.03 −0.17
𝜕W (1)
[
]
−0.22 0.86 0.39
=
0.58 −0.38 −0.13
W (2) = W (2) − 𝛼
b(1) = b(1) − 𝛼
[
]
⎡ −0.3 ⎤ ⎡0.19⎤
0.11 0.23
𝜕J
⎢−0.06⎥ = ⎢0.25⎥
=
−
0.25
⎢
⎥ ⎢
⎥
0.36
𝜕W (2)
⎣−0.51⎦ ⎣0.49⎦
[
]
[
]
𝜕J
= 0.18 −0.49 0.42 − 0.25 0.04 −0.03 −0.11
𝜕b(1) [
]
= 0.19 −0.48 0.45
[
]
[
] [
]
b(2) = b(2) − 𝛼 𝜕b𝜕J(2) = 0.28 − 0.25 0.31 = 0.36 .
Using these new values, a new forward propagation for a new epoch starts as the training
continues.
325
327
Appendix C
Back Propagation in a CNN
Figure C.1 shows the ANN used in the example presented in Chapter 7. This fully-connected
ANN has (8x4) synapses between the inputs and hidden layer (i.e. |W (1) | = 32) and (1x4) for bias
(i.e. |B(1) | = 4), and (4x2) neurons between the hidden layer and the output layer (i.e. |W (2) | = 8)
and (1x2) for bias (i.e. |B(2) | = 2).
Recall from Figure 7.14c that the inputs and weights for the ANN layer generated by the UL are
[
]
X = 1.03 0.53 0.53 1.03 1.05 1.05 0.55 1.05 .
Since the example input is in the shape of an O (Figure 7.7), the expected ANN outputs will be
Y1 = 1 and Y2 = 0. In matrix form, these outputs are
[ ]
Y= 1 0 .
Weights and Biases of ANN
To illustrate the back propagation process in a CNN, suppose that initially the ANN weights and
biases are selected randomly as follows:
⎡−0.4 −0.1 0.1 0.5 ⎤
⎢ 0.5 −0.1 0.3 0.5 ⎥
⎥
⎢
⎢−0.5 0.2 0.2 0.4 ⎥
⎢ 0.2 −0.4 0.2 0.4 ⎥
W (1) = ⎢
⎥
⎢ 0.3 −0.2 0.3 −0.3⎥
⎢ 0.2 0.4 0.5 0.4 ⎥
⎢ 0.2 −0.1 0.1 0.2 ⎥
⎥
⎢
⎣−0.4 0.5 0.3 −0.3⎦
B(1) =
[
]
0.1 0.3 0.4 −0.2
⎡0.4 0.2 ⎤
⎢
⎥
0.3 −0.2⎥
W (2) = ⎢
⎢0.5 −0.3⎥
⎢0.1 0.3 ⎥
⎣
⎦
[
]
B(2) = 0.1 0.1 .
Back Propagation Starting at ANN
Suppose we obtained the following values after forward and back propagation through the
fully-connected ANN (intermediate computations are skipped for brevity, but they are the same as
those presented in Chapter 2):
[
]
Y = 0.53 0.41
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
328
Appendix C Back Propagation in a CNN
Figure C.1 Fully-connected ANN used in CNN
for this example with (8x4) synapses between
the inputs and hidden layer (i.e. |W (1) | = 32)
and (1x4) for bias (i.e. |B(1) | = 4), and (4x2)
neurons between the hidden layer and the
output layer (i.e. |W (2) | = 8) and (1x2) for bias
(i.e. |B(2) | = 2)
⎡ −0.4 −0.8 0.12 0.52 ⎤
⎢ 0.51 −0.11 0.31 0.49 ⎥
⎥
⎢
⎢−0.52 0.22 0.2 0.36 ⎥
⎢ 0.2 −0.35 0.21 0.4 ⎥
W (1) = ⎢
⎥
⎢ 0.33 −0.2 0.27 −0.31⎥
⎢ 0.19 0.37 0.41 0.42 ⎥
⎢ 0.2 −0.17 0.17 0.23 ⎥
⎥
⎢
⎣−0.31 0.51 0.32 −0.31⎦
⎡0.4 0.2 ⎤
⎢
⎥
0.3 −0.2⎥
W (2) = ⎢
⎢0.5 −0.3⎥
⎢0.1 0.3 ⎥
⎣
⎦
[
]
𝛿 (2) = 0.8 −0.7 0.13 −0.11 .
Back Propagation through CL
For 𝛿 C representing the delta of the convolution layer and . ∗ an element-wise multiplication, we
have:
T
⎡ −0.4 −0.8 0.12 0.52 ⎤
⎢ 0.51 −0.11 0.31 0.49 ⎥
⎥
⎢
⎢−0.52 0.22 0.2 0.36 ⎥
[
] ⎢ 0.2 −0.35 0.21 0.4 ⎥
𝛿 C = 𝛿 (2) ⋅ W (1)T . ∗ f ′ (C) = 0.8 −0.7 0.13 −0.11 ⎢
⎥
⎢ 0.33 −0.2 0.27 −0.31⎥
⎢ 0.19 0.37 0.41 0.42 ⎥
⎢ 0.2 −0.17 0.17 0.23 ⎥
⎥
⎢
⎣−0.31 0.51 0.32 −0.31⎦
[
]
.∗ 1 1 1 1 1 1 1 1
[
]
= −0.7 0.03 −0.07 0.03 0.11 −0.01 0.01 0.01 .
Mapping from PL to CL
Based on each maximum selected by the PL previously during the forward propagation, we now
map the maximums onto their corresponding input regions generated at the CL as shown in
Figures C.2a and b.
For Filter 1, we have
(
)
[
]
[
]
𝜕J
= IC=1 ⋅ 𝛿 (2) = −1 1 1 −1 ⋅ (−0.07) = 0.07 −0.07 −0.07 0.07
𝜕F
( 1 )1
[
]
[
]
𝜕J
= IC=5 ⋅ 𝛿 (2) = −1 1 −1 −1 ⋅ (0.03) = −0.03 0.03 −0.03 −0.03
𝜕F1 2
Appendix C Back Propagation in a CNN
(a)
Figure C.2
(
(
(b)
Mapping maximum values into corresponding regions in input image for Filters 1 (a) and 2 (b)
𝜕J
𝜕F1
𝜕J
𝜕F1
)
[
]
[
]
= IC=5 ⋅ 𝛿 (2) = −1 1 −1 −1 ⋅ (−0.07) = 0.07 −0.07 0.07 0.07
)3
[
]
[
]
= IC=9 ⋅ 𝛿 (2) = −1 1 1 −1 ⋅ (0.03) = −0.03 0.03 0.03 −0.03 ,
4
th
where
( IC=i
) means the input matrix region used during the i convolution, as shown in Figure C.2a,
𝜕J
and 𝜕F
represents the partial derivative of J with respect to filter F1 at the ith pooling output.
1
i
)
4 (
∑
[
] [
]
𝜕J
𝜕J
=
= 0.07 −0.07 −0.07 0.07 + −0.03 0.03 −0.03 −0.03
𝜕F1
𝜕F
1 i
i=1
[
] [
]
0.07 −0.07 0.07 0.07 + −0.03 0.03 0.03 −0.03
[
]
= 0.08 −0.08 0 0.08
4
∑
𝜕J
=
𝛿 C = (−0.07) + (0.03) + (−0.07) + (0.03) = −0.08.
𝜕B(1) n=1 i
Similarly, for Filter 2, we have
(
)
[
]
[
]
𝜕J
= IC=2 ⋅ 𝛿 (2) = 1 −1 −1 1 ⋅ (0.11) = 0.11 −0.11 −0.11 0.11
𝜕F2 1
(
)
[
]
[
]
𝜕J
= IC=6 ⋅ 𝛿 (2) = 1 −1 −1 1 ⋅ (0.01) = 0.01 −0.01 −0.01 0.01
𝜕F
( 2 )2
[
]
[
]
𝜕J
= IC=7 ⋅ 𝛿 (2) = 1 −1 1 1 ⋅ (0.01) = 0.01 −0.01 0.01 0.01
𝜕F
( 2 )3
[
]
[
]
𝜕J
= IC=6 ⋅ 𝛿 (2) = 1 −1 −1 1 ⋅ (0.01) = 0.01 −0.01 −0.01 0.01 ,
𝜕F2 4
329
330
Appendix C Back Propagation in a CNN
th
where
) means the input matrix region used during the i convolution, as shown in Figure C.2b,
( IC=i
𝜕J
represents the partial derivative of J with respect to filter F2 at the ith pooling output.
and 𝜕F
2
i
)
4 (
∑
[
] [
]
𝜕J
𝜕J
=
= 0.11 −0.11 −0.11 0.11 + 0.01 −0.01 −0.01 0.01
𝜕F2
𝜕F2 i
i=1
[
] [
]
0.01 −0.01 0.01 0.01 + 0.01 −0.01 −0.01 0.01
[
]
= 0.14 −0.14 −0.12 0.14
8
∑
𝜕J
=
𝛿 C = (0.11) + (0.01) + (0.01) + (0.01) = 0.14.
𝜕B(2) n=5 i
New Weights and Biases for the CNN
Let the learning rate be 𝛼 = 0.5 (this rate is typically too high for realistic cases but is chosen here
to demonstrate the onvergence of weights):
[
]
[
]
𝜕J
= 1 −1 −1 1 − 0.5 0.08 −0.08 0 0.08
F1(new) = F1 − 𝛼
𝜕F1
[
]
= 0.96 −0.96 −1 0.96
[
]
[
]
𝜕J
= −1 1 1 −1 − 0.5 0.14 −0.14 0.12 0.14
F2(new) = F2 − 𝛼
𝜕F2
[
]
= −1.07 1.07 1.06 −1.07
[ ]
[
] [
]
𝜕J
B(1)
= B(1) − 𝛼 (1) = 0.1 − 0.25 0.08 = 0.14
(new)
𝜕B
[ ]
[
] [
]
𝜕J
= B(2) − 𝛼 (2) = 0.2 − 0.5 0.14 = 0.13 ,
B(2)
(new)
𝜕B
where subscript (new) used in the left-hand sides of the equations indicates the updated values of
the filters and biases, whereas the current values are in the right-hand sides. After modifying the
filters and biases, training continues with a new forward propagation and error computation using
the ANN outputs, followed by a back propagation to modify the filters, and so on.
331
Appendix D
Back Propagation Through Time in an RNN
D.1
Back Propagation in an RNN
In this appendix, we present expressions for computing the partial derivatives of loss with respect
to RNN parameters of h, W, Ω, V, c and b during the back propagation through time used in RNNs
as discussed in Chapter 10. Let us first present the definitions of RNN parameters from Chapter 10:
ht = Θh (W ht−1 + U xt + b)
(D.1)
Ωt = V ht + c
(D.2)
ŷ t = Θŷ (Ωt )
(D.3)
||
∑
Lt = −
yot(i) ln(̂yt(i) ),
(D.4)
i=1
where Θh is realized as tanh and Θy as softmax. In order to minimize the loss L, its gradients
with respect to U, W, V, b and c must be computed. Recall from Eq. (10.7) in Chapter 10 that
cross-entropy loss is defined as
 ||
∑
∑
yot(i) ln(̂yt(i) ).
L=−
t=1 i=1
Using Eq. (D.4), let us now compute the partial derivative of Lt with respect to the ith element of
Ωt (i.e. Ωt(i) ) as follows:
𝜕Lt
𝜕 ŷ
𝜕
1
=−
(yot(i) ln(̂yt(i) )) = −yot(i)
.
𝜕Ωt(i)
𝜕Ωt(i)
ŷ t(i) 𝜕Ωt(i)
(D.5)
From Chapter 10, we have the definition of softmax as
⎡ eΩt,1 ⎤
⎢ ∑|| Ωt,j ⎥
⎢ j=1 e ⎥
⎢ eΩt,2 ⎥
⎥
⎢∑
|| Ω
eΩt
ŷ t = Θŷ = softmax(Ωt ) = ⎢ j=1 e t,j ⎥ = ∑||
,
⎥
⎢
eΩt,j
j=1
⎥
⎢
⋮
⎥
⎢ Ω
t,||
⎥
⎢ e
⎢ ∑|| eΩt,j ⎥
⎦
⎣ j=1
(D.6)
where || is the size of an alphabet , and Ωt,i is the ith element of the Ωt matrix. We can see in
Eqn. (D.6) that ŷ t elements have values between 0 and 1. It shows that elements of raw output Ωt ,
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
332
Appendix D Back Propagation Through Time in an RNN
whose values may be negative, are replaced with their power over e (i.e. eΩt,i ) and divided by the
summation of all raw output values as powers of e. The values for ŷ t vector, therefore, are probabilities with values between 0 and 1.
Since softmax is a IR|| → IR|| mapping function, the derivative with respect to Ωt is as follows:
𝜕 ŷ t(1)
𝜕 ŷ t(1) ⎤
⎡ 𝜕 ŷ t(1)
⎥
⎢ 𝜕Ω · · · 𝜕Ω · · · 𝜕Ω
t(1)
t(i)
t(||) ⎥
⎢
⎥
⎢
⋮
⎥
⎢
𝜕 ŷ t(j)
𝜕 ŷ t(j) ⎥
𝜕 ŷ t ⎢ 𝜕 ŷ t(j)
=⎢
···
···
⎥,
Ωt
𝜕Ωt(i)
𝜕Ωt(||) ⎥
⎢ 𝜕Ωt(1)
⎥
⎢
⋮
⎥
⎢
𝜕 ŷ t(||)
𝜕 ŷ t(||) ⎥
⎢ 𝜕 ŷ t(||)
···
···
⎥
⎢
𝜕Ωt(i)
𝜕Ωt(||) ⎦
⎣ 𝜕Ωt(1)
(D.7)
where the number of rows and the columns are the alphabet size of ‖|. Let us now compute the
value for the partial derivative of ŷ t with respect to Ωt for arbitrary i and j values in the matrix shown
in Eq. (D.7):
(
)
𝜕 ŷ t(j)
𝜕
eΩt(j)
=
,
𝜕Ωt(i)
𝜕Ωt(i) ∑|| eΩt(k)
k=1
where i and j represent different elements of the matrix in Eq. (D.7). Let us consider the case where
i = j:
∑||
𝜕 ŷ t(j)
eΩt(j) k=1 eΩt(k) − eΩt(j) eΩt(i)
=
(∑
)2
𝜕Ωt(i)
|| Ωt(k)
e
k=1
eΩt(j)
( ∑||
= ∑||
eΩt(k)
k=1
k=1
eΩt(k) − eΩt(i)
∑||
eΩt(k)
k=1
)
eΩt(j)
(
= ∑||
eΩt(k)
k=1
eΩt(i)
1 − ∑||
k=1
)
eΩt(k)
= ŷ t(j) ( 1 − ŷ t(i) ).
(D.8)
Let us now consider the case of i ≠ j:
𝜕 ŷ t(j)
−eΩt(j) eΩt(i)
−eΩt(j)
eΩt(i)
= (
=
)
∑
∑
|| Ωt(k)
|| Ωt(k)
∑|| Ωt(k) 2
𝜕Ωt(i)
e
e
e
k=1
k=1
k=1
= −̂yt(j) ŷ t(i) .
(D.9)
For step t and i = j, plugging Eq. (D.8) into Eq. (D.5) we have
||
∑
𝜕Lt
1
ŷ t(j) (1 − ŷ t(i) ) = −yot(i) (1 − ŷ t(i) ) = −yot(i) + yot(i) ŷ t(i) ,
= − yot(j)
̂
𝜕Ωt(i)
y
t(j)
j=1
where only one element is evaluated under the summation (i.e. i = j case). For step t and i ≠ j,
plugging Eq. (D.9) into Eq. (D.5) we have
||
||
∑
∑
𝜕Lt
1
= − yot(j)
(−̂yt(j) ŷ t(i) ) =
yot(j) ŷ t(i) .
̂
𝜕Ωt(i)
y
t(j)
j=1
j=1,j≠i
D.1 Back Propagation in an RNN
Therefore, for the general case we have
||
∑
𝜕Lt
= −yot(i) + yot(i) ŷ t(i) +
yot(j) ŷ t(i) ,
𝜕Ωt(i)
j=1,j≠i
where we can include yot(i) ŷ t(i) into the summation since j ≠ i is missing in it, and change the summation limits as follows:
||
∑
𝜕Lt
= −yot(i) +
yot(j) ŷ t(i) ,
𝜕Ωt(i)
j=1
where ŷ t(i) is independent of index j and hence can be taken out of the summation:
||
∑
𝜕Lt
o
̂
= −yt(i) + yt(i) yot(j) ,
𝜕Ωt(i)
j=1
where the summation of all one-hot encoded values of the expected RNN outputs is equal to one:
𝜕Lt
= ŷ t(i) − yot(i) ,
𝜕Ωt(i)
which can be used to generalize the loss for  steps:

∑
𝜕L
=
(̂yt − yot ).
𝜕Ωt
t=1
(D.10)
Let us now consider the partial derivative of L with respect to V:



𝜕L ∑ 𝜕Lt ∑ 𝜕Lt 𝜕 ŷ t 𝜕Ωt ∑ 𝜕Lt 𝜕Ωt
=
=
=
.
𝜕V
𝜕V
𝜕Ωt 𝜕V
𝜕 ŷ t 𝜕Ωt 𝜕V
t=1
t=1
t=1
From Eq. (D.2), we know that 𝜕Ωt ∕𝜕V = h†t . Substituting Eq. (D.10), we obtain

𝜕L ∑
=
(̂yt − yot ) ⋅ h†t .
𝜕V
t=1
(D.11)
For the partial derivative of L with respect to c, we have



𝜕L ∑ 𝜕Lt ∑ 𝜕Lt 𝜕 ŷ t 𝜕Ωt ∑ 𝜕L 𝜕Ωt
=
=
=
.
𝜕c
𝜕c
𝜕Ωt 𝜕c
𝜕 ŷ t 𝜕Ωt 𝜕c
t=1
t=1
t=1
From Eq. (D.2), we have 𝜕Ωt ∕𝜕c = 1. Substituting Eq. (D.10), we get

𝜕L ∑
(̂yt − yot ).
=
𝜕c
t=1
(D.12)
We now consider the partial derivative of L with respect to W. Since W is related to the previous
hidden state, we use recursion in Eq. (D.1) for t = 1, … , 4:
h1 = Θh (W h0 + U x1 + b) h2 = Θh (W h1 + U x2 + b)
h3 = Θh (W h2 + U x3 + b) h4 = Θh (W h3 + U x4 + b).
By placing previous hidden-state terms into the current hidden state, we get
h1 = Θh (W h0 + U x1 + b)
h2 = Θh (W (Θh (W h0 + U x1 + b)) + U x2 + b)
333
334
Appendix D Back Propagation Through Time in an RNN
h3 = Θh (W (Θh (W (Θh (W h0 + U x1 + b)) + U x2 + b)) + U x3 + b)
h4 = Θh (W Θh (W Θh (W Θh (Wh0 + Ux1 + b) + Ux2 + b) + Ux3 + b) + Ux4 + b),
where Θh = tanh. Let us obtain the partial derivatives of Lt for t = 1, · · · , 4:
𝜕L 𝜕 ŷ 1 𝜕h1
𝜕L1
= 1
𝜕W
𝜕 ŷ 1 𝜕h1 𝜕W
𝜕L 𝜕 ŷ 2 𝜕h2 𝜕h1 𝜕L2 𝜕 ŷ 2 𝜕h2
𝜕L2
= 2
+
𝜕W
𝜕 ŷ 2 𝜕h2 𝜕h1 𝜕W
𝜕 ŷ 2 𝜕h2 𝜕W
𝜕L3
𝜕L 𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕h1 𝜕L3 𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕L3 𝜕 ŷ 3 𝜕h3
= 3
+
+
𝜕W
𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕h1 𝜕W
𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕W
𝜕 ŷ 3 𝜕h3 𝜕W
𝜕L 𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕h3
𝜕L4
= 4
+
+
𝜕W
𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕W
𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕W
𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕W
+
𝜕L4 𝜕 ŷ 4 𝜕h4
.
𝜕 ŷ 4 𝜕h4 𝜕W
We can then generalize the derivative terms for step t as follows:
( t−1
)
t
∑
𝜕Lt
𝜕hk
𝜕Lt 𝜕 ŷ t ∏ 𝜕hj+1
=
.
𝜕W
𝜕W
𝜕 ŷ t 𝜕ht j=k 𝜕hj
k=1
Let us now present the formula for  steps:
)
( t
( t−1
)

∑
∑ 𝜕Lt 𝜕 ŷ t ∏
𝜕hj+1
𝜕hk
𝜕L
=
.
𝜕W
𝜕W
𝜕 ŷ t 𝜕ht j=k 𝜕hj
t=1
k=1
(D.13)
Finally, we insert the following partial derivatives into above formulas:
𝜕Lt
= (̂yt − yot )
𝜕Ωt
𝜕Ωt
From Eq. (D.2):
= V†
𝜕ht
𝜕hj+1
= W † (1 − h2j+1 )
From Eq. (D.1):
𝜕hj
𝜕hk
= (1 − h2k ) hk−1 .
From Eq. (D.1):
𝜕W
Inserting Eqs. (D.14), (D.15), (D.16) and (D.17) into Eq. (D.13), we have
)
( t
( t−1
)

∑
∑
∏
𝜕L
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) hk−1 .
𝜕W
t=1
k=1
j=k
From Eq. (D.10):
(D.14)
(D.15)
(D.16)
(D.17)
(D.18)
Now, let us compute the partial derivative of L with respect to U. Similar to the case of W in
Eq. (D.18), for U we have
)
( t
( t−1
)

𝜕hk
𝜕L ∑ ∑ 𝜕Lt 𝜕 ŷ t ∏ 𝜕hj+1
=
.
(D.19)
𝜕U
𝜕U
𝜕 ŷ t 𝜕ht j=k 𝜕hj
t=1
k=1
From Eq. (D.1) we have the partial derivative of hk with respect to U as
𝜕hk
= (1 − h2k ) xk .
𝜕U
(D.20)
D.2 Embedding Layer
Inserting Eqs. (D.14), (D.15), (D.16) and (D.20) into Eq. (D.19), we obtain
( t
( t−1
)
)

∏
𝜕L ∑ ∑
o
†
†
2
2
=
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) xk .
𝜕U
t=1
k=1
j=k
(D.21)
Continuing in the same manner, the gradient of L with respect to b is calculated similarly to
Eq. (D.19) as follows:
)
( t
( t−1
)

𝜕hk
𝜕L ∑ ∑ 𝜕Lt 𝜕 ŷ t ∏ 𝜕hj+1
=
.
(D.22)
𝜕b
𝜕b
𝜕 ŷ t 𝜕ht j=k 𝜕hj
t=1
k=1
Inserting Eqs. (D.14), (D.15), (D.16) and (D.20) into Eq. (D.22), we obtain
( t
( t−1
))

∏
𝜕L ∑ ∑
o
†
†
2
=
(̂yt − yt ) V
W (1 − hj+1 )
.
𝜕b
t=1
k=1
j=k
D.2
(D.23)
Embedding Layer
When an embedding layer is used in an RNN as discussed in Section 10.5 in Chapter 10, we have
xt = E ⋅ xto ,
(D.24)
where xto s the one-hot-encoded input vector and E is the embedding weight matrix. For an RNN
with an embedding layer, all forward propagation equations given above in Eqs. (D.1) to (D.4) are
still valid. However, we now need to include the partial derivative of L with respect to E since the
weights in E will be updated as training goes from one epoch to another. The only equation that is
related to E is Eq. (D.24). For steps t = 1, · · · , 4, we have the partial derivative of L with respect to
E as
𝜕L1
𝜕L 𝜕 ŷ 1 𝜕h1 𝜕x1
= 1
𝜕E
𝜕 ŷ 1 𝜕h1 𝜕x1 𝜕E
𝜕L2
𝜕L 𝜕 ŷ 2 𝜕h2 𝜕h1 𝜕x1 𝜕L2 𝜕 ŷ 2 𝜕h2 𝜕x2
= 2
+
𝜕E
𝜕 ŷ 2 𝜕h2 𝜕h1 𝜕x1 𝜕E
𝜕 ŷ 2 𝜕h2 𝜕x2 𝜕E
𝜕L 𝜕 ŷ 3 𝜕h3 𝜕h1 𝜕x1 𝜕L3 𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕x2 𝜕L3 𝜕 ŷ 3 𝜕h3 𝜕x3
𝜕L3
= 3
+
+
𝜕E
𝜕 ŷ 3 𝜕h3 𝜕h1 𝜕x1 𝜕E
𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕x2 𝜕E
𝜕 ŷ 3 𝜕h3 𝜕x3 𝜕E
𝜕L4
𝜕L 𝜕 ŷ 4 𝜕h4 𝜕h1 𝜕x1 𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕h2 𝜕x2 𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕x3
= 4
+
+
𝜕E
𝜕 ŷ 4 𝜕h4 𝜕h1 𝜕x1 𝜕E
𝜕 ŷ 4 𝜕h4 𝜕h2 𝜕x2 𝜕E
𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕x3 𝜕E
+
𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕x4
.
𝜕 ŷ 4 𝜕h4 𝜕x4 𝜕E
𝜕h
Let us now expand 𝜕h t as follows:
k
𝜕L1 𝜕 ŷ 1 𝜕h1 𝜕x1
𝜕L1
=
𝜕E
𝜕 ŷ 1 𝜕h1 𝜕x1 𝜕E
𝜕L2
𝜕L2 𝜕 ŷ 2 𝜕h2 𝜕h1 𝜕x1 𝜕L2 𝜕 ŷ 2 𝜕h2 𝜕x2
=
+
𝜕E
𝜕 ŷ 2 𝜕h2 𝜕h1 𝜕x1 𝜕E
𝜕 ŷ 2 𝜕h2 𝜕x2 𝜕E
𝜕L3 𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕h1 𝜕x1 𝜕L3 𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕x2 𝜕L3 𝜕 ŷ 3 𝜕h3 𝜕x3
𝜕L3
=
+
+
𝜕E
𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕h1 𝜕x1 𝜕E
𝜕 ŷ 3 𝜕h3 𝜕h2 𝜕x2 𝜕E
𝜕 ŷ 3 𝜕h3 𝜕x3 𝜕E
335
336
Appendix D Back Propagation Through Time in an RNN
𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕x1 𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕x2
𝜕L4
=
+
𝜕E
𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕x1 𝜕E
𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕h2 𝜕x2 𝜕E
+
𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕x3 𝜕L4 𝜕 ŷ 4 𝜕h4 𝜕x4
+
.
𝜕 ŷ 4 𝜕h4 𝜕h3 𝜕x3 𝜕E
𝜕 ŷ 4 𝜕h4 𝜕x4 𝜕E
For step t, we then have the following:
(
)
t
t−1
𝜕Lt ∑ 𝜕Lt 𝜕 ŷ t ∏ 𝜕hj+1
𝜕hk 𝜕xk
=
.
̂
𝜕E
𝜕xk 𝜕E
𝜕h
𝜕h
𝜕
y
t j=k
j
t
k=1
For  steps we have
)
( t
(
)

t−1
𝜕hk 𝜕xk
𝜕L ∑ ∑ 𝜕Lt 𝜕 ŷ t ∏ 𝜕hj+1
=
.
𝜕E
𝜕xk 𝜕E
𝜕hj
𝜕 ŷ t 𝜕ht j=k
t=1
k=1
(D.25)
Recall that we have the individual partial derivatives from above as
𝜕Lt
= (̂yt − yot )
𝜕 ŷ t
𝜕 ŷ t
= V†
𝜕ht
𝜕hj+1
𝜕hj
= W † (1 − h2j+1 )
𝜕hk
= (1 − h2k ) U
𝜕xk
𝜕xk
= xko .
𝜕E
By substituting them into Eq. (D.25), we obtain
( t
)
t−1 (

)
∏
𝜕L ∑ ∑
o
†
†
2
2
o
(̂yt − yt ) V
W (1 − hj+1 ) (1 − hk ) U xk .
=
𝜕E
t=1
k=1
j=k
(D.26)
337
Appendix E
Back Propagation Through Time in an RNN with Attention
E.1
Back Propagation in an RNN-Att
In this appendix, for an RNN utilizing an attention mechanism, we present computing the partial
derivatives of loss with respect to parameters of h, , W, , Ω, V, c and b during a back propagation
through time used in RNN-Att in Chapter 12.
ht = Θh (W ht−1 + U xt + b)
(E.1)
t,i = hi ⋅ ht for i = 1, … t
(E.2)
et,i
for i = 1, … t
t,k
k=0 e
softmax(t,i ) = ∑t
t =
t
∑
softmax(t,k ) ⋅ hk
(E.3)
(E.4)
k=0
Ωt = V ⋅ t + c
(E.5)
ŷ t = Θŷ (Ωt )
(E.6)
||
∑
Lt = −
yot(i) ln(̂yt(i) ),
(E.7)
i=1
where Θh and Θy represent tanh in Eq. (E.1) and softmax in Eq. (E.6), respectively. The partial
derivative of ŷ t with respect to Ωt presented in Eq. (D.5) in Appendix D is also valid for an RNN-Att
as follows:
𝜕Lt
= ŷ t − yot .
(E.8)
𝜕Ωt
Since weight matrix V stays consistent during time steps, we can perform differentiation at each
time step and then aggregate the results. From Eqs. (E.8), (E.4) and (E.5), we have



𝜕L ∑ 𝜕Lt ∑ 𝜕Lt 𝜕 ŷ t 𝜕Ωt ∑
=
=
=
(̂yt − yot ) ⋅ h†t .
̂
𝜕V
𝜕V
𝜕Ω
𝜕V
𝜕
y
t
t
t=1
t=1
t=1
(E.9)
For c, from Eqs. (E.8) and (E.5), we have
T
T
T
𝜕L ∑ 𝜕Lt ∑ 𝜕Lt 𝜕 ŷ t 𝜕Ωt ∑
=
=
=
(̂yt − yot ).
̂
𝜕c
𝜕c
𝜕Ω
𝜕c
𝜕
y
t
t
t=1
t=1
t=1
(E.10)
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
338
Appendix E Back Propagation Through Time in an RNN with Attention
The partial derivative of L with respect to W has to be computed through recursion as shown in
Eq. (E.1). We start from the partial derivative of L1 with respect to W at t = 1:
𝜕L 𝜕 ŷ 1 𝜕Ω1 𝜕1 𝜕h1
𝜕L1
= 1
.
𝜕W
𝜕 ŷ 1 𝜕Ω1 𝜕1 𝜕h1 𝜕W
For t = 2, W includes a dependency on h1 and h2 , which yields
(
)
𝜕L2 𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h1
𝜕L2
=
𝜕W
𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h1 𝜕W
(
)
𝜕L2 𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕L2 𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕h1
+
+
.
𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕W
𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕h1 𝜕W
Similarly, at t = 3, W includes h1 , h2 and h3 dependencies as follows:
(
)
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h1
𝜕L3
=
𝜕W
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h1 𝜕W
(
)
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕h1
+
+
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕W
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕h1 𝜕W
(
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2
+
+
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕W
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2 𝜕W
)
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2 𝜕h1
+
.
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2 𝜕h1 𝜕W
For t = 4, the partial derivative of L4 will includes terms of h1 , h2 , h3 and h4 :
(
)
𝜕L4
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h1
=
𝜕W
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h1 𝜕W
(
)
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕h1
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕W
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕h1 𝜕W
(
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕W
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2 𝜕W
)
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2 𝜕h1
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2 𝜕h1 𝜕W
(
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕W
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕W
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕h1
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕W
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕W
)
.
Let us now group the common terms:
) (
)
((
𝜕L4 𝜕L4 𝜕 ŷ 4 𝜕Ω4
𝜕4 𝜕h2 𝜕4 𝜕h2 𝜕h1
𝜕4 𝜕h1
=
+
+
𝜕W 𝜕 ŷ 4 𝜕Ω4 𝜕4
𝜕h1 𝜕W
𝜕h2 𝜕W
𝜕h2 𝜕h1 𝜕W
(
)
𝜕4 𝜕h3 𝜕4 𝜕h3 𝜕h2 𝜕4 𝜕h3 𝜕h2 𝜕h1
+
+
+
𝜕h3 𝜕W
𝜕h3 𝜕h2 𝜕W
𝜕h3 𝜕h2 𝜕h1 𝜕W
))
(
𝜕4 𝜕h4 𝜕4 𝜕h4 𝜕h3 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕h1
.
+
+
+
+
𝜕h4 𝜕W
𝜕h4 𝜕h3 𝜕W
𝜕h4 𝜕h3 𝜕h2 𝜕W
𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕W
E.1 Back Propagation in an RNN-Att
Let us introduce a more compact version of this formula:
( 4 m
(
))
m−1
𝜕L4 𝜕L4 𝜕 ŷ 4 𝜕Ω4 ∑ ∑ 𝜕4 ∏ 𝜕hj+1 𝜕hk
=
.
𝜕W 𝜕 ŷ 4 𝜕Ω4 𝜕4 m=1 k=1 𝜕hm j=k 𝜕hj
𝜕W
Let us generalize Eq. (E.11) for  steps:


∑
∑
𝜕Lt
𝜕Lt 𝜕 ŷ t 𝜕Ωt
𝜕L
=
=
𝜕W
𝜕W
𝜕 ŷ t 𝜕Ωt 𝜕t
t=1
t=1
(E.11)
( t m
))
(
∑ ∑ 𝜕t m−1
∏ 𝜕hj+1 𝜕hk
m=1 k=1
𝜕hm j=k 𝜕hj
𝜕W
.
(E.12)
From Eqs. (E.1)–(E.7), we obtain the following partial derivatives:
𝜕Lt 𝜕 ŷ t
= (̂yt − yot )
𝜕 ŷ t 𝜕Ωt
(E.13)
𝜕Ωt
= V†
𝜕Zt
(E.14)
𝜕t
= t,m
𝜕hm
(E.15)
𝜕hj+1
𝜕hj
= W † (1 − h2j+1 )
(E.16)
𝜕hk
= (1 − h2k ) hk−1 .
(E.17)
𝜕W
Substituting Eqs. (E.13) to (E.17) into the partial derivative of L with respect to W in Eq. (E.12),
we have
)
( t m

m−1
∑
∑∑
∏
(
)
𝜕L
o
†
†
2
2
(̂yt − yt ) V
t,m W (1 − hj+1 ) (1 − hk ) hk−1 .
(E.18)
=
𝜕W
t=1
m=1 k=1
j=k
Now, let us compute the partial derivative of L with respect to U. Similarly to the case of W in
Eq. (E.12) above, for U we have
( t m
))
(


m−1
𝜕L ∑ 𝜕Lt ∑ 𝜕Lt 𝜕 ŷ t 𝜕Ωt ∑ ∑ 𝜕t ∏ 𝜕hj+1 𝜕hk
=
=
.
(E.19)
𝜕U
𝜕U
𝜕U
𝜕 ŷ t 𝜕Ωt 𝜕t m=1 k=1 𝜕hm j=k 𝜕hj
t=1
t=1
Substituting Eqs. (E.13) to (E.17) into the partial derivative of L with respect to U in Eq. (E.19),
we obtain
)
( t m

m−1
∑∑
∏
(
)
𝜕L ∑
o
†
†
2
2
(̂yt − yt ) V
t,m W (1 − hj+1 ) (1 − hk ) xk
.
(E.20)
=
𝜕U
t=1
m=1 k=1
j=k
The partial derivative of L with respect to b is calculated in a similar manner. As in the case of U
in Eq. (E.19), for b we have
( t m
(
))


m−1
𝜕L ∑ 𝜕Lt ∑ 𝜕Lt 𝜕 ŷ t 𝜕Ωt ∑ ∑ 𝜕t ∏ 𝜕hj+1 𝜕hk
=
=
.
(E.21)
𝜕b
𝜕b
𝜕b
𝜕 ŷ t 𝜕Ωt 𝜕t m=1 k=1 𝜕hm j=k 𝜕hj
t=1
t=1
We substitute Eqs. (E.13) to (E.17) into Eq. (E.21), and obtain
( t m
)
m−1

∑∑
∏
(
)
𝜕L ∑
o
†
†
2
2
=
(̂yt − yt ) V
t,m W (1 − hj+1 ) (1 − hk ) .
𝜕b
t=1
m=1 k=1
j=k
(E.22)
339
340
Appendix E Back Propagation Through Time in an RNN with Attention
E.2
Embedding Layer
For the utilization of an embedding layer in an RNN-Att as discussed in Section 12.5 in Chapter 12,
we have
xt = xto E,
(E.23)
where xto s the one-hot-encoded input vector, and E is the embedding weight matrix. For an
RNN-Att with embedding layer, all forward propagation equations given above in Eqs. (E.1) to
(E.7) are still valid. We now need to compute the partial derivative of L with respect to E for
updating the weights in E during training. Eq. (D.24) needs to be considered in the computation of
gradients. For steps t = 1, · · · , 4, we have the partial derivatives of L with respect to E as follows:
𝜕L1
𝜕L 𝜕 ŷ 1 𝜕Ω1 𝜕1 𝜕h1 𝜕x1
= 1
𝜕E
𝜕 ŷ 1 𝜕Ω1 𝜕1 𝜕h1 𝜕x1 𝜕E
𝜕L2
=
𝜕E
𝜕L3
=
𝜕E
𝜕L4
=
𝜕E
(
)
𝜕L2 𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h1 𝜕x1
𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h1 𝜕x1 𝜕E
(
)
𝜕L2 𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕x2 𝜕L2 𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕h1 𝜕x1
+
+
𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕x2 𝜕E
𝜕 ŷ 2 𝜕Ω2 𝜕2 𝜕h2 𝜕h1 𝜕x1 𝜕E
(
)
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h1 𝜕x1
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h1 𝜕x1 𝜕E
(
)
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕x2 𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕h1 𝜕x1
+
+
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕x2 𝜕E
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h2 𝜕h1 𝜕x1 𝜕E
(
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕x3 𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2 𝜕x2
+
+
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕x3 𝜕E
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2 𝜕x2 𝜕E
)
𝜕L3 𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2 𝜕h1 𝜕x1
+
𝜕 ŷ 3 𝜕Ω3 𝜕3 𝜕h3 𝜕h2 𝜕h1 𝜕x1 𝜕E
(
)
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h1
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h1 𝜕E
)
(
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕h1
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕E
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h2 𝜕h1 𝜕E
(
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕E
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2 𝜕E
)
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2 𝜕h1
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h3 𝜕h2 𝜕h1 𝜕E
(
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕E
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕E
𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕h1
+
+
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕E
𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕E
)
.
E.2 Embedding Layer
Let us now rearrange the terms in the equation above:
(
𝜕L4 𝜕L4 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h1 𝜕x1 𝜕4 𝜕h2 𝜕4 𝜕h2 𝜕h1 𝜕x1
=
+
+
𝜕E 𝜕 ŷ 4 𝜕Ω4 𝜕4 𝜕h1 𝜕x1 𝜕E
𝜕h2 𝜕W
𝜕h2 𝜕h1 𝜕x1 𝜕E
+
𝜕4 𝜕h3 𝜕x3 𝜕4 𝜕h3 𝜕h2 𝜕x2 𝜕4 𝜕h3 𝜕h2 𝜕h1 𝜕x1
+
+
𝜕h3 𝜕x3 𝜕E
𝜕h3 𝜕h2 𝜕x2 𝜕E
𝜕h3 𝜕h2 𝜕h1 𝜕x1 𝜕E
𝜕4 𝜕h4 𝜕x4 𝜕4 𝜕h4 𝜕h3 𝜕x3 𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕x2
+
+
𝜕h4 𝜕x4 𝜕E
𝜕h4 𝜕h3 𝜕x3 𝜕E
𝜕h4 𝜕h3 𝜕h2 𝜕x2 𝜕E
)
𝜕4 𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕x1
+
.
𝜕h4 𝜕h3 𝜕h2 𝜕h1 𝜕x1 𝜕E
+
A more compact form of the partial derivative the becomes
( 4 m
))
(
m−1
𝜕L4 𝜕L4 𝜕 ŷ 4 𝜕Ω4 ∑ ∑ 𝜕4 ∏ 𝜕hj+1 𝜕hk 𝜕xk
=
.
𝜕E 𝜕 ŷ 4 𝜕Ω4 𝜕4 m=1 k=1 𝜕hm j=k 𝜕hj
𝜕xk 𝜕E
For  steps, we have

𝜕L ∑ 𝜕Lt 𝜕 ŷ t 𝜕Ωt
=
𝜕E
𝜕 ŷ t 𝜕Ωt 𝜕t
t=1
( t m
))
(
∑ ∑ 𝜕t m−1
∏ 𝜕hj+1 𝜕hk 𝜕xk
m=1 k=1
𝜕hm j=k 𝜕hj
𝜕xk 𝜕E
.
(E.24)
We have the individual partial derivatives from above as
𝜕Lt
= (̂yt − yot )
𝜕Ωt
𝜕Ωt
= V†
𝜕Zt
𝜕t
= t,m
𝜕hm
𝜕hj+1
𝜕hj
= W † (1 − h2j+1 )
𝜕hk
= (1 − h2k ) U
𝜕xk
𝜕xk
= xko .
𝜕E
By substituting these into Eq. (E.24), we have the partial derivative of L with respect to E as
)
( t m

m−1
∑∑
∏
(
)
𝜕L ∑
o
†
†
2
2
o
=
(̂yt − yt ) V
t,m W (1 − hj+1 ) (1 − hk ) U xk
.
(E.25)
𝜕E
t=1
m=1 k=1
j=k
341
343
Bibliography
1 A. Pinar Saygin, I. Cicekli, and V. Akman, “Turing test: 50 years later,” Minds and Machines,
vol. 10, no. 4, pp. 463–518, 2000.
2 A. M. Turing, “Computing machinery and intelligence (october 1950),” Mind LIX (236):
433–460, doi:10.1093/mind/LIX.236.433, ISSN 0026-4423, 2009.
3 J. McCarthy, “Professor John McCarthy – jmc.stanford.edu,” http://jmc.stanford.edu/,
[Accessed 06-Jul-2023].
4 N. J. Nilsson, “Shakey the robot (technical note 323, artificial intelligence center, sri international, menlo park, california),” 1984.
5 W. Saletan, “Chess bump: The triumphant teamwork of humans and computers,” Slate.
Archived from the original on, vol. 13, 2007.
6 “33 Industries Other Than Auto That Driverless Cars Could Turn Upside Down cbinsights.com,” https://www.cbinsights.com/research/13-industries-disrupted-driverlesscars/, [Accessed 06-Jul-2023].
7 “AI and Robotics, Tesla,” https://www.tesla.com/AI, [Accessed 09-Jul-2023].
8 A. Sherstinsky, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,” Physica D: Nonlinear Phenomena, vol. 404, p. 132306, 2020.
9 S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision:
A survey,” ACM Computing Surveys (CSUR), vol. 54, no. 10s, pp. 1–41, 2022.
10 “Gboard will now use AI to auto-suggest emoji, stickers and GIFs based on the conversation,
91mobiles.com – 91mobiles.com,” https://www.91mobiles.com/hub/gboard-auto-suggest-emojigif-stickers-ai-details/, [Accessed 09-Jul-2023].
11 W. Bellamy-III, “Airbus to Use AI in Designing FCAS Flight Control Software - Avionics
International - aviationtoday.com,” https://www.aviationtoday.com/2019/06/28/airbus-use-aidesigning-fcas-flight-control-software/, [Accessed 09-Jul-2023].
12 P. A. Langen, J. S. Katz, G. Dempsey, and J. Pompano, “Remote monitoring of high-risk
patients using artificial intelligence,” 1994, US Patent 5,357,427.
13 A. Saribudak, Y. Dong, J. Hsieh, and M. U. Uyar, “Modeling tumor growth for kidney cancer based on nuclei clusters of pathology slides,” International Journal of Engineering and
Technology, vol. 8, no. 5, p. 375, 2016.
14 “Bloomberg - Are you a robot?” https://www.bloomberg.com/news/articles/2016-09-20/
microsoft-develops-ai-to-help-cancer-doctors-&hellip;-treatments#xj4y7vzkg, [Accessed
09-Jul-2023].
15 D. A. Hashimoto, G. Rosman, D. Rus, and O. R. Meireles, “Artificial intelligence in surgery:
Promises and perils,” Annals of Surgery, vol. 268, no. 1, p. 70, 2018.
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
344
Bibliography
16 Á. Takács, I. Rudas, D. Bösl, and T. Haidegger, “Highly automated vehicles and self-driving
cars [industry tutorial],” IEEE Robotics & Automation Magazine, vol. 25, no. 4, pp. 106–112,
2018.
17 N. O’Mahony, S. Campbell, L. Krpalkova, D. Riordan, J. Walsh, A. Murphy, and C. Ryan,
“Deep learning for visual navigation of unmanned ground vehicles: A review,” in 2018 29th
Irish Signals and Systems Conference (ISSC). IEEE, 2018, pp. 1–6.
18 J. Kusyk, M. U. Uyar, K. Ma, E. Samoylov, R. Valdez, J. Plishka, S. E. Hoque, G. Bertoli, and
J. Boksiner, “Artificial intelligence and game theory controlled autonomous uav swarms,”
Evolutionary Intelligence, vol. 14, pp. 1775–1792, 2021.
19 J. Zou, S. Gundry, J. Kusyk, M. Ü. Uyar, and C. S. Sahin, “3D genetic algorithms for underwater sensor networks,” International Journal of Ad Hoc and Ubiquitous Computing, vol. 13,
no. 1, pp. 10–22, 2013.
20 F. Azam, M. Sharif, M. Yasmin, and S. Mohsin, “Artificial intelligence based techniques for
earthquake prediction: A review,” Sci Int, vol. 26, no. 4, pp. 1495–1502, 2014.
21 T. Marwala and E. Hurwitz, Artificial intelligence and economic theory: Skynet in the market.
Springer, 2017, vol. 1.
22 N. Aletras, D. Tsarapatsanis, D. Preoţiuc-Pietro, and V. Lampos, “Predicting judicial decisions
of the European Court of Human Rights: A natural language processing perspective,” PeerJ
Computer Science, vol. 2, p. e93, 2016.
23 K. L. Priddy and P. E. Keller, Artificial neural networks: An introduction. SPIE press, 2005,
vol. 68.
24 T. Ma, F. Wang, J. Cheng, Y. Yu, and X. Chen, “A hybrid spectral clustering and deep neural
network ensemble algorithm for intrusion detection in sensor networks,” Sensors, vol. 16,
no. 10, p. 1701, 2016.
25 M. Egmont-Petersen, D. de Ridder, and H. Handels, “Image processing with neural networks –
a review,” Pattern Recognition, vol. 35, no. 10, pp. 2279–2301, 2002.
26 G. Bylinsky, “Computers that learn by doing,” Fortune, vol. 128, no. 5, 1993.
27 J. Pradeep, E. Srinivasan, and S. Himavathi, “Diagonal based feature extraction for handwritten character recognition system using neural network,” in 2011 3rd international conference
on electronics computer technology, vol. 4. IEEE, 2011, pp. 364–368.
28 A. R. Naik and S. Pathan, “Weather classification and forecasting using back propagation
feed-forward neural network,” International Journal of Scientific and Research Publications,
vol. 2, no. 12, pp. 1–3, 2012.
29 B. Egeli, M. Ozturan, and B. Badur, “Stock market prediction using artificial neural networks,”
Decision Support Systems, vol. 22, pp. 171–185, 2003.
30 D. C. Montgomery, E. A. Peck, and G. G. Vining, Introduction to linear regression analysis.
John Wiley & Sons, 2021.
31 “Neural networks demystified, Welch Labs,” https://www.youtube.com/watch?v=bxe2T-V8XRs,
2020, [Accessed 28-Jul-2023].
32 N. Burkart and M. F. Huber, “A survey on the explainability of supervised machine learning,”
Journal of Artificial Intelligence Research, vol. 70, pp. 245–317, 2021.
33 M. Usama, J. Qadir, A. Raza, H. Arif, K.-L. A. Yau, Y. Elkhatib, A. Hussain, and
A. Al-Fuqaha, “Unsupervised machine learning for networking: Techniques, applications
and research challenges,” IEEE Access, vol. 7, pp. 65 579–65 615, 2019.
34 L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” Journal
of Artificial Intelligence Research, vol. 4, pp. 237–285, 1996.
35 J. S. Judd, Neural network design and the complexity of learning. MIT press, 1990, vol. 2.
Bibliography
36 “Normalizing inputs for an artificial neural network,” https://www.baeldung.com/cs/
normalizing-inputs-artificial-neural-network, [Accessed 18-Sep-2023].
37 K. Jing and J. Xu, “A survey on neural network language models,” arXiv preprint
arXiv:1906.03591, 2019.
38 H. Mavrodiev, “London bike sharing dataset,” https://www.kaggle.com/datasets/hmavrodiev/
london-bike-sharing-dataset, 2019, [Accessed 28-Jul-2023].
39 J. Kamruzzaman and R. A. Sarker, “Forecasting of currency exchange rates using ANN: A case
study,” in International Conference on Neural Networks and Signal Processing, 2003. Proceedings
of the 2003, vol. 1. IEEE, 2003, pp. 793–797.
40 “Stochastic oscillator,” https://en.wikipedia.org/wiki/Stochastic_oscillator, [Accessed
15-Jul-2024].
41 J. Chen, “Rate of change definition, formula, and importance,” https://www.investopedia.com/
terms/r/rateofchange.asp, 2024, [Accessed 15-Jul-2024].
42 M. Lesinski, T. Hortobágyi, T. Muehlbauer, A. Gollhofer, and U. Granacher, “Effects of balance training on balance performance in healthy older adults: A systematic review and
meta-analysis,” Sports Medicine, vol. 45, pp. 1721–1738, 2015.
43 C. Mitchell, “Williams %r: Definition, formula, uses, and limitations,” https://www
.investopedia.com/terms/w/williamsr.asp, 2021, [Accessed 15-Jul-2024].
44 J. L. Person, A complete guide to technical trading tactics: How to profit using pivot points,
candlesticks & other indicators. John Wiley & Sons, 2004, vol. 217.
45 https://mebfaber.com/2013/06/07/qtaa-paper-update-manage-your-risk-4/, 2013, [Accessed
28-Jul-2023].
46 S. G. Anderson, “Momentum indicates stock price strength,” Investopedia, 2021, [Accessed
27-Oct-2023].
47 “Exponential Moving Average,” https://www.fidelity.com/learning-center/trading-investing/
technical-analysis/technical-indicator-guide/ema, [Accessed 15-Jul-2024].
48 https://corporatefinanceinstitute.com/resources/knowledge/trading-investing/bollinger-bands/,
[Accessed 28-Jul-2023].
49 “Exponential Moving Average,” https://www.fidelity.com/learning-center/trading-investing/
technical-analysis/technical-indicator-guide/bollinger-bands, [Accessed 15-Jul-2024].
50 “Simple Moving Average Explanation and Trading Strategies — TradingSim –
tradingsim.com,” https://tradingsim.com/blog/simple-moving-average, [Accessed 15-Jul-2024].
51 “High frequency data,” https://en.wikipedia.org/wiki/High_frequency_data, [Accessed
15-Jul-2024].
52 “2023 Alzheimer’s disease facts and figures,” Alzheimer’s & Dementia, vol. 19, no. 4,
pp. 1598–1695, 2023. [Online]. Available: https://alz-journals.onlinelibrary.wiley.com/doi/
abs/10.1002/alz.13016
53 “2021 Alzheimer’s disease facts and figures,” Alzheimer’s and Dementia: The Journal of the
Alzheimer’s Association, vol. 17, pp. 327–406, 2021.
54 I. Arevalo-Rodriguez, N. Smailagic, M. R. i Figuls, A. Ciapponi, E. Sanchez-Perez,
A. Giannakou, O. L. Pedraza, X. B. Cosp, and S. Cullum, “Mini-mental state examination (MMSE) for the detection of alzheimer’s disease and other dementias in people
with mild cognitive impairment (MCI),” Cochrane Database of Systematic Reviews, no. 3,
2015.
55 T. Lee and H. Lee, “Prediction of Alzheimer’s disease using blood gene expression data,”
Scientific Reports, vol. 10, no. 1, p. 3485, 2020.
345
346
Bibliography
56 “Alzheimer’s Disease Neuroimaging Initiative.” ADNI, adni.loni.usc.edu/. Accessed 15
Aug. 2023.
57 C. Hinrichs et al. “Predictive markers for AD in a multi-modality framework: An analysis of
MCI progression in the ADNI population.” NeuroImage, vol. 55, no. 2, Mar. 2011, pp. 574–89,
https://doi.org/10.1016/j.neuroimage.2010.10.081. Accessed 30 Dec. 2019.
58 Karteek Popuri et al. “Using machine learning to quantify structural MRI neurodegeneration
patterns of Alzheimer’s disease into dementia score: Independent validation on 8,834 images
from ADNI, AIBL, OASIS, and MIRIAD databases.” Human Brain Mapping, vol. 41, no. 14,
Wiley, July 2020, pp. 4127–47, https://doi.org/10.1002/hbm.25115. Accessed 3 May 2023.
59 ‘V. Mantzavinos and A. Alexiou, “Biomarkers for Alzheimer’s disease diagnosis.”
Current Alzheimer Research, vol. 14, no. 11, Oct. 2017, https://doi.org/10.2174/
1567205014666170203125942.
60 A. Saribudak, A. A. Subick, N. H. Kim, J. A. Rutta, and M. Ü. Uyar, “Gene expressions, hippocampal volume loss, and mmse scores in computation of progression and pharmacologic
therapy effects for Alzheimer’s disease,” IEEE/ACM Transactions on Computational Biology
and Bioinformatics, vol. 17, no. 2, pp. 608–622, 2018.
61 R. Cavoretto, “Adaptive LOOCV-based kernel methods for solving time-dependent BVPs,”
Applied Mathematics and Computation, vol. 429, p. 127228, 2022.
62 D. Vidaurre, C. Bielza, and P. Larranaga, “A survey of l1 regression,” International Statistical
Review, vol. 81, no. 3, pp. 361–387, 2013.
63 S. Bird, E. Klein, and E. Loper, Natural language processing with Python: Analyzing text with
the natural language toolkit. “O’Reilly Media, Inc.”, 2009.
64 A. Yadav, A. Patel, and M. Shah, “A comprehensive review on resolving ambiguities in natural
language processing,” AI Open, vol. 2, pp. 85–92, 2021.
65 B. Charbuty and A. Abdulazeez, “Classification based on decision tree algorithm for machine
learning,” Journal of Applied Science and Technology Trends, vol. 2, no. 01, pp. 20–28, 2021.
66 S. P. Lende and M. Raghuwanshi, “Question answering system on education acts using NLP
techniques,” in 2016 world conference on futuristic trends in research and innovation for social
welfare (Startup Conclave). IEEE, 2016, pp. 1–6.
67 M. Allahyari, S. Pouriyeh, M. Assefi, S. Safaei, E. D. Trippe, J. B. Gutierrez, and K. Kochut,
“Text summarization techniques: A brief survey,” arXiv preprint arXiv:1707.02268, 2017.
68 M. H. Miller, J. Muthuswamy, and R. E. Whaley, “Mean reversion of Standard & Poor’s 500
index basis changes: Arbitrage-induced or statistical illusion?” The Journal of Finance, vol. 49,
no. 2, pp. 479–513, 1994.
69 D. Boyd, S. Golder, and G. Lotan, “Tweet, tweet, retweet: Conversational aspects of retweeting on twitter,” in 2010 43rd Hawaii international conference on system sciences. IEEE, 2010,
pp. 1–10.
70 https://neurdiness.wordpress.com/2018/05/17/deep-convolutional-neural-networks-as-modelsof-the-visual-system-qa/, [Accessed 28-Jul-2023].
71 K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image
recognition,” arXiv preprint arXiv:1409.1556, 2014.
72 Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document
recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
73 C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and
A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on
computer vision and pattern recognition, 2015, pp. 1–9.
Bibliography
74 B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM
Journal on Control and Optimization, vol. 30, no. 4, pp. 838–855, 1992.
75 J. Denker, W. Gardner, H. Graf, D. Henderson, R. Howard, W. Hubbard, L. D. Jackel, H. Baird,
and I. Guyon, “Neural network recognizer for hand-written zip code digits,” Advances in
Neural Information Processing Systems, vol. 1, 1988.
76 R. Johnson and T. Zhang, “Effective use of word order for text categorization with convolutional neural networks,” arXiv preprint arXiv:1412.1058, 2014.
77 M. Hashemi, “Enlarging smaller images before inputting into convolutional neural network:
Zero-padding vs. interpolation,” Journal of Big Data, vol. 6, no. 1, pp. 1–13, 2019.
78 “What is OCR (optical character recognition)? — Definition from TechTarget – techtarget.com,” https://www.techtarget.com/searchcontentmanagement/definition/
OCR-optical-character-recognition, [Accessed 15-Jul-2023].
79 “MNIST handwritten digit database, Yann LeCun, Corinna Cortes and Chris
Burges – yann.lecun.com,” http://yann.lecun.com/exdb/mnist/, [Accessed 15-Jul-2023].
80 “mnist | TensorFlow Datasets – tensorflow.org,” https://www.tensorflow.org/datasets/catalog/
mnist/, [Accessed 15-Jul-2023].
81 “Number of voice assistants in use worldwide 2019–2024 | Statista – statista.com,” https://
www.statista.com/statistics/973815/worldwide-digital-voice-assistant-in-use/, [Accessed
15-Jul-2023].
82 L. Wyse, “Audio spectrogram representations for processing with convolutional neural
networks,” arXiv preprint arXiv:1706.09559, 2017.
83 E. O. Brigham, The fast Fourier transform and its applications. Prentice-Hall, Inc., 1988.
84 “spectrogram - Wiktionary – en.wiktionary.org,” https://en.wiktionary.org/wiki/spectrogram,
[Accessed 15-Jul-2023].
85 B. Gao and L. Pavel, “On the properties of the softmax function with application in game
theory and reinforcement learning,” arXiv preprint arXiv:1704.00805, 2017.
86 D. Hakkani-Tür, G. Tür, A. Celikyilmaz, Y.-N. Chen, J. Gao, L. Deng, and Y.-Y. Wang,
“Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM.” in Interspeech,
2016, pp. 715–719.
87 H. Kameoka, W.-C. Huang, K. Tanaka, T. Kaneko, N. Hojo, and T. Toda, “Many-to-many voice
transformer network,” IEEE/ACM Transactions on Audio, Speech, and Language Processing,
vol. 29, pp. 656–670, 2020.
88 S. Bagui, D. Nandi, S. Bagui, and R. J. White, “Machine learning and deep learning for
phishing email classification using one-hot encoding,” Journal of Computer Science, vol. 17,
pp. 610–623, 2021.
89 P. Cerda, G. Varoquaux, and B. Kégl, “Similarity encoding for learning with dirty categorical
variables,” Machine Learning, vol. 107, no. 8–10, pp. 1477–1494, 2018.
90 K. W. Church, “Word2vec,” Natural Language Engineering, vol. 23, no. 1, pp. 155–162, 2017.
91 J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,”
in Proceedings of the 2014 conference on empirical methods in natural language processing
(EMNLP), 2014, pp. 1532–1543.
92 A. Coates and A. Y. Ng, “The importance of encoding versus training with sparse coding and
vector quantization,” in Proceedings of the 28th international conference on machine learning
(ICML-11), 2011, pp. 921–928.
93 Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks
with noisy labels,” Advances in Neural Information Processing Systems, vol. 31, 2018.
347
348
Bibliography
94 O. Hrinchuk, V. Khrulkov, L. Mirvakhabova, E. Orlova, and I. Oseledets, “Tensorized embedding layers for efficient model compression,” arXiv preprint arXiv:1901.10787, 2019.
95 B. Hanin, “Which neural net architectures give rise to exploding and vanishing gradients?”
Advances in Neural Information Processing Systems, vol. 31, 2018.
96 M.-T. Luong, “Effective approaches to attention-based neural machine translation,” arXiv
preprint arXiv:1508.04025, 2015.
97 A. Pinar Saygin, I. Cicekli, and V. Akman, “Turing test: 50 years later,” Minds and Machines,
vol. 10, no. 4, pp. 463–518, 2000.
98 M. L. Mauldin, “Chatterbots, tinymuds, and the Turing test: Entering the Loebner prize competition,” in AAAI, vol. 94, 1994, pp. 16–21.
99 K. Brush and J. Scardina, “What is a chatbot?” https://www.techtarget.com/ searchcustomerexperience /definition/chatbot, techtarget.com, 2024, [Accessed 8-June-2024].
100 G. Caldarini, S. Jaf, and K. McGarry, “A literature survey of recent advances in chatbots,”
2022, mDPI: 41. arXiv:2201.06657. doi:10.3390/info13010041.
101 G. Neubig, “Neural machine translation and sequence-to-sequence models: A tutorial,” arXiv
preprint arXiv:1703.01619, 2017.
102 D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align
and translate,” 2016. [Online]. Available: https://arxiv.org/abs/1409.0473
103 M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural
machine translation,” 2015. [Online]. Available: https://arxiv.org/abs/1508.04025
104 D. E. Goldberg, Genetic algorithms in search, optimization, and machine Learning.
Addison-Wesley, 1989.
105 R. Eberhart and J. Kennedy, “A new optimizer using particle swarm theory,” in MHS’95.
Proceedings of the sixth international symposium on micro machine and human science. Ieee,
1995, pp. 39–43.
106 R. Storn, “Differrential evolution – a simple and efficient adaptive scheme for global optimization over continuous spaces,” Technical report, International Computer Science Institute, vol. 11,
1995.
107 J. R. Koza, Genetic programming: On the programming of computers by means of natural selection. MIT Press, 1992.
108 J. H. Holland, Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. MIT press, 1992.
109 W. Stephenson, “Number – the language of science. by Tobias Dantzig. London: George
Allen & Unwin, Ltd., 1930. Large crown 8vo. pp. 260,” Journal of Mental Science, vol. 77,
no. 319, pp. 843–843, 1931.
110 “Movie Theater Food Nutrition Guide,” https://www.pinterest.com/pin/497155246383350429/,
[Accessed 02-Aug-2024].
111 F. Vaezi, S. Sadjadi, and A. Makui, “A robust knapsack based constrained portfolio optimization,” International Journal of Engineering, vol. 33, no. 5, pp. 841–851, 2020.
112 “Knapsack problem - Wikipedia – en.wikipedia.org,” https://en.wikipedia.org/wiki/Knapsack_
problem, [Accessed 09-Jul-2023].
113 Ş. Uyar and H. T. Uyar, “A critical look at dynamic multi-dimensional knapsack problem generation,” in Applications of evolutionary computing: EvoWorkshops 2009, Tübingen, Germany,
April 15–17, 2009. Proceedings. Springer, 2009, pp. 762–767.
114 T. H. Cormen, C. E. Leiserson, R. Rivest, and C. Stein, “Introduction to algorithms,” London:
MIT Press. Section, vol. 25, pp. 636–40, 2009.
Bibliography
115 M. Alp Ertem and N. Buyurgan, “An auction-based framework for resource allocation in disaster relief,” Journal of Humanitarian Logistics and Supply Chain Management, vol. 1, no. 2,
pp. 170–188, 2011.
116 V. A. Vasyanin, “Problem of distribution and routing of transport blocks with mixed attachments and its decomposition,” Journal of Automation and Information Sciences, vol. 47, no. 2,
2015.
117 S. Snow, “Pentagon successfully tests world’s largest micro drone swarm,” https://www
.militarytimes.com/news/pentagon-congress/2017/01/09/pentagon-successfully-tests-worlds-largest-micro-drone-swarm/, [Accessed 09-Jul-2023].
118 “Rantizo Drones for Agriculture | Drone Application and Services – rantizo.com,” https://www
.rantizo.com/, [Accessed 09-Jul-2023].
119 J. Zou, J. Kusyk, M. Ü. Uyar, S. Gundry, and C. S. Sahin, “Bio-inspired and voronoi-based
algorithms for self-positioning autonomous mobile nodes,” in MILCOM 2012-2012 IEEE Military Communications Conference. IEEE, 2012, pp. 1–6.
120 Q. Yao, J. Xiong, P. Liu, H. Zhou, G. Rao, J. Deng, S. Pan, and J. Wang, “Determination of the
phase diagrams of the nd2fe14b–pr2fe14b isopleth,” Journal of Alloys and Compounds, vol.
633, pp. 229–232, 2015.
121 G. Zaccone, Python parallel programming cookbook. Packt Publishing Ltd, 2015.
122 K. L. Hoffman, M. Padberg, G. Rinaldi et al., “Traveling salesman problem,” Encyclopedia of
Operations Research and Management Science, vol. 1, pp. 1573–1578, 2013.
123 M. Grotschel, M. Junger, and G. Reinelt, “Optimal control of plotting and drilling machines: A
case study,” Mathematical Methods of Operations Research, vol. 35, no. 1, pp. 61–84, 1991.
124 K. Reagan, “The Travelling Salesman’s Power,” https://rjlipton.wordpress.com/2012/04/22/thetravelling-salesmans-power/, 2021, [Accessed 28-Jul-2023].
125 L. Jacobson, “Applying a genetic algorithm to the traveling salesman problem,” http://www
.theprojectspot.com/tutorial-post/applying-a-genetic-algorithm-to-the-travelling-salesmanproblem/5/, 2012, [Accessed 28-Jul-2023].
126 “Optimal County Tour,” https://www.math.uwaterloo.ca/tsp/county/index.html#:~:
text=The%20optimal%20tour%20of%203%2C100,to%20make%20along%20the%20way., 2015,
[Accessed 28-Jul-2023].
127 “Travelling salesman problem,” https://en.wikipedia.org/wiki/Travelling_salesman_problem,
2023, [Accessed 28-Jul-2023].
128 T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to algorithms. MIT
press, 2022.
129 J. A. Chisman, “The clustered traveling salesman problem,” Computers & Operations Research,
vol. 2, no. 2, pp. 115–119, 1975.
130 J. R. Current and D. A. Schilling, “The covering salesman problem,” Transportation Science,
vol. 23, no. 3, pp. 208–213, 1989.
131 S. Varun Kumar and R. Panneerselvam, “A study of crossover operators for genetic algorithms
to solve VRP and its variants and new sinusoidal motion crossover operator,” International
Journal of Computational Intelligence Research, vol. 13, no. 7, pp. 1717–1733, 2017.
132 G. Beni and J. Wang, “Swarm intelligence in cellular robotics,” in Systems Proceedings of
NATO Advanced Workshop on Robots and Biological Systems, vol. 102, 1989.
133 M. Gairing, B. Monien, and K. Tiemann, “Selfish routing with incomplete information,”
in Proceedings of the seventeenth annual ACM symposium on parallelism in algorithms and
architectures, 2005, pp. 203–212.
349
350
Bibliography
134 J. Sneyd, G. Theraula, E. Bonabeau, J.-L. Deneubourg, and N. R. Franks, Self-organization in
biological systems. Princeton University Press, 2001.
135 H. Shayeghi, M. Mahdavi, and A. Bagheri, “Discrete PSO algorithm-based optimization of
transmission lines loading in TNEP problem,” Energy Conversion and Management, vol. 51,
no. 1, pp. 112–121, 2010.
136 H. Wang, H. Sun, C. Li, S. Rahnamayan, and J.-s. Pan, “Diversity enhanced particle swarm
optimization with neighborhood search,” Information Sciences, vol. 223, pp. 119–135, 2013.
137 E. S. Peer, F. van den Bergh, and A. P. Engelbrecht, “Using neighbourhoods with the guaranteed convergence PSO,” in Proceedings of the 2003 IEEE Swarm Intelligence Symposium. SIS’03
(Cat. No. 03EX706). IEEE, 2003, pp. 235–242.
138 Y. H. Robinson, M. Rajaram et al., “Energy-aware multipath routing scheme based on particle
swarm optimization in mobile ad hoc networks,” The Scientific World Journal, vol. 2015, 2015.
139 N. Keerthipriya and R. Latha, “Adaptive cluster formation in MANET using particle swarm
optimization,” in 2015 3rd international conference on signal processing, communication and
networking (ICSCN). IEEE, 2015, pp. 1–7.
140 D. Van der Merwe and A. P. Engelbrecht, “Data clustering using particle swarm optimization,” in The 2003 Congress on Evolutionary Computation, 2003. CEC’03., vol. 1. IEEE, 2003,
pp. 215–220.
141 S. Kok and C. Sandrock, “Locating and characterizing the stationary points of the extended
Rosenbrock function,” Evolutionary Computation, vol. 17, no. 3, pp. 437–453, 2009.
142 S. Gundry, J. Zou, M. U. Uyar, C. S. Sahin, and J. Kusyk, “Differential evolution-based
autonomous and disruption tolerant vehicular self-organization in MANETs,” Ad Hoc
Networks, vol. 25, pp. 454–471, 2015.
143 X. Li, L. Xu, H. Wang, J. Song, and S. X. Yang, “A differential evolution-based routing
algorithm for environmental monitoring wireless sensor networks,” Sensors, vol. 10, no. 6,
pp. 5425–5442, 2010.
144 R. Gämperle, S. D. Müller, and P. Koumoutsakos, “A parameter study for differential evolution,” Advances in Intelligent Systems, Fuzzy Systems, Evolutionary Computation, vol. 10, no. 10,
pp. 293–298, 2002.
145 K. K. Mandal, V. Haldar, and N. Chakraborty, “Comparison of different variants of differential
evolution applied to short-term economic generation scheduling of hydrothermal systems,” in
2010 Conference Proceedings IPEC. IEEE, 2010, pp. 836–841.
146 V. Plagianakos, D. Tasoulis, and M. N. Vrahatis, “A review of major application areas of differential evolution,” Advances in Differential Evolution, pp. 197–238, 2008.
147 F. Stenger, “Numerical methods based on Whittaker cardinal or sinc functions,” Siam Review,
vol. 23, no. 2, pp. 165–224, 1981.
148 J. S. Rohl, “A note on Backus–Naur form,” The Computer Journal, vol. 10, no. 4, pp. 336–337,
1968.
149 D. D. McCracken and E. D. Reilly, “Backus–naur form (bnf),” in Encyclopedia of Computer
Science, 2003, pp. 129–131.
150 M. O’Neill, C. Ryan, M. O’Neil, and C. Ryan, Grammatical evolution. Springer, 2003.
151 M. O’Neill, F. Leahy, and A. Brabazon, “Grammatical swarm: A variable-length particle swarm
algorithm,” in Swarm Intelligent Systems. Springer, 2006, pp. 59–74.
152 M. O’Neill, A. Brabazon, M. Nicolau, S. M. Garraghy, and P. Keenan, “𝜋 grammatical evolution,” in Genetic and evolutionary computation – GECCO 2004: Genetic and Evolutionary
Computation Conference, Seattle, WA, USA, June 26–30, 2004. Proceedings, Part II. Springer,
2004, pp. 617–629.
Bibliography
153 E. Medvet, “Hierarchical grammatical evolution,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2017, pp. 249–250.
154 E. Ferrante, E. Duéñez-Guzmán, A. E. Turgut, and T. Wenseleers, “Geswarm: Grammatical
evolution for the automatic synthesis of collective behaviors in swarm robotics,” in Proceedings
of the 15th annual conference on genetic and evolutionary computation, 2013, pp. 17–24.
155 E. Kita, H. Sugiura, Y. Zuo, and T. Mizuno, “Application of grammatical evolution to stock
price prediction,” Computer Assisted Methods in Engineering and Science, vol. 24, no. 1,
pp. 67–81, 2017.
156 A. Brabazon and M. O’Neill, “Credit classification using grammatical evolution,” Informatica,
vol. 30, no. 3, 2006.
157 D. Howard, “Bioinformatics with evolutionary computation,” in Real world applications of
computational intelligence. Springer, 2005, pp. 245–281.
158 J. Kusyk, M. U. Uyar, and C. S. Sahin, “Survey on evolutionary computation methods for
cybersecurity of mobile ad hoc networks,” Evolutionary Intelligence, vol. 10, pp. 95–117, 2018.
159 S. Şen and J. A. Clark, “A grammatical evolution approach to intrusion detection on mobile ad
hoc networks,” in Proceedings of the second ACM conference on wireless network security, 2009,
pp. 95–102.
160 Y. T. Lim, P. C. Cheng, J. A. Clark, and P. Rohatgi, “Policy evolution with grammatical
evolution,” in Simulated evolution and learning: 7th International Conference, SEAL 2008,
Melbourne, Australia, December 7–10, 2008. Proceedings 7. Springer, 2008, pp. 71–80.
161 J. S. Rohl, “Grammatical evolution and the Santa Fe trail problem,” pp. 10–19, 2010.
351
353
Index
a
activation function 8
algorithmic trading 3, 51, 55, 57
Alzheimer’s disease 67
ADNI 68
gene expressions 68
LOOCV 69
M coefficients 68
MCI 67
Mild cognitive impairment 67
mini mental state exam 68
MMSE 68
Alzheimer’s disease prognosis 4
ANN 2, 7, 23
activation function 8
algorithmic trading 3, 51, 55, 57, 58
Alzheimer’s disease 68
Alzheimer’s disease prognosis 4
architecture 3, 7
back propagation 14, 28, 30
back testing 58
bias 17
bike rental example 3, 41
computational complexity 18
forward propagation 12, 28
hidden layers 25
inputs 24
learning rate 29, 32
LOOCV 69
loss 17
natural language processing 83
neuron 8
NLP 4, 83
NLP ANN architecture 88
normalization of inputs and outputs 19
normalized inputs and oututs 31
outputs 24
rectilinear unit 9
RELU 9
sigmoid 9
sleep-study example 3, 25
synapses 10
tick data 88
training 11, 12, 18
training with matrices 26
updating weights 17, 29
weights 10
artificial intelligence 1, 2
artificial neural network 2, 7, 23
attenion scores 193
attention 191
b
back propagation 14, 28, 30, 157
back testing 58
Backus–Naur form 307
bias 17
bike rental example 41
bio-inspired computation 225
BNF 307
Bollinger bands 54, 55
c
chatbot 4, 175, 179, 183
chatbot implementaion 179, 183
chromosome 226
Machine Learning and AI with Simple Python and Matlab Scripts: Courseware for Non-computing Majors, First Edition. M. Ümit Uyar.
© 2025 The Institute of Electrical and Electronics Engineers, Inc. Published 2025 by John Wiley & Sons, Inc.
Companion website: www.wiley.com/go/UyarSimplePythonandMatlabScripts1e
354
Index
chromosome for swarm flight 257
CNN 2, 4, 99
ANN 99, 106
architecture 102
computational cost 108
convolutional layer 99, 104
OCR 4, 115, 119, 125
optical character recognition 115, 119, 125
pooling layer 99, 106
RELU 99, 104
spectrogram 133
speech recognition 4, 133, 134, 138, 144
training 100
unrolling layer 99, 106
codon 226
convolutional layer 99
convolutional neural network 2, 99
architecture 4
crossover 226, 228
crossover, partially matched 284
d
DE 298
decoder 153
dietary menu selection 235, 240, 242, 245
differential evolution 298
drone flight control 251
drone swarm 251
e
EMA 54, 55
embedding layer 165, 200
encoder 153
evolutionary algorithms 2, 5
BNF 307
differrential evolution 298
grammatical evolution 5, 306
particle swarm optimization 5, 291
PSO 5
exponential moving average 54
g
GA 4, 226
child 228
chromosome 226
chromosome for swarm flight 257
codon 226
crossover 226, 228
dietary menu selection 235, 240, 242, 245
drone flight control 251
fitness 255
fitness function 227
flight control 251
force-based fitness 255
grammatical evolution 307
knapsack problem 235, 237
mutation 226, 229
offspring 228
parent 228
population 226
prtially matched crossover 284
route optimization 274, 284
swarm flight fitness function 254
travelling salesman problem 5, 273, 274, 278
two-point crossover 229
UAV control fitness function 254
UAV flight control 5
UAV swarm 251
gene expressions 68
generative AI 2
genetic algorithm 4, 226
grammatical evolution 5, 306
h
hidden state 153
i
imitation game
175
k
keywords 86
knapsack problem 235, 237
f
financial technical indicators 51
fitness 255
fitness function 227, 254
force-based fitness 255
forward propagation 12, 28, 157
l
learning rate 29, 32
leave-one-out cross validation 69
London bike rental example 41
LOOCV 69
Index
m
machine translation 4, 209, 211, 216
many-to-many single cell 210
Matlab script
algorithmic trading 63
Alzheimer’s disease prognosis 75
bike rental example 46
dietary menu selection 245
machine translation 216
NLP implementation 92
optical character recognition 125
RNN chatbot 183
route optimization 284
sleep-study example 37
speech recognition 144
UAV swarm control 264
momentum 53, 55
mutation 226, 229
n
natural language processing 4
neuron 8
NLP 4, 83
inputs and outputs 85
keywords 86
tick data 88
tweets 86
normalization of inputs and outputs 19
o
OCR 4, 115
one-hot encoding 156
optical character recognition 4, 115, 119, 125
p
particle swarm optimization 291
pooling layer 99
PSO 291
Python script
algorithmic trading 59
Alzheimer’s disease prognosis 71
bike rental example 41
dietary menu selection 242
machine translation 211
NLP implementation 89
optical character recognition 119
RNN chatbot 179
route optimization 279
sleep-study example 34
speech recognition 138
UAV swarm control 260
r
rectilinear unit 9
recurrent neural network 2, 4, 151
RNN 2, 4, 151
architecture 4
attention 191
back propagation 157
chatbot 4, 175, 179, 183
decoder 153
embedding layer 165
encoder 153
forward propagation 157
hidden layer 164
hidden state 153
learning rate 158
loss function 157
many-to-many architecture 175
multiple hidden layers 164
one-hot encoding 156
one-to-one single cell 153
softmax 153, 155
training 157
training with embedding layer 167
updating weights 158
RNN with attention 4
RNN-Att 4, 191
attention 191
attention scores 193
context vector 192
decoder 191
embedding layer 200
encoder 191
learning rate 195
loss 194
machine translation 4, 209
many-to-many single cell 210
one-to-one single cell 191
softmax 192, 194
training 193
updating weights 195, 201
route optimization 274, 279, 284
355
356
Index
s
sigmoid 9, 10
simple moving average 52
sleep-study example 25
SMA 52, 55
softmax 153, 155, 194
spectrogram 133
speech recognition 4, 133, 134, 138,
144
synapses 10
travelling salesman problem 5, 273, 274
chromosome 278
turing test 1, 175
tweets 86
two-point crossover 229
u
UAV 251
UAV flight control 5
UAV swarm control 260, 264
unmanned aerial vehicle 251
t
tick data 57, 58, 88
training 11, 12, 18, 193
w
weights
10
WILEY END USER LICENSE AGREEMENT
Go to www.wiley.com/go/eula to access Wiley’s ebook EULA.