Department of Electrical And Electronic Engineering
Khulna University of Engineering & Technology
Khulna – 9203, Bangladesh
An improved deep learning based approach of steering angle
prediction for autonomous vehicle
Supervised by:
Submitted by:
Dr. Md. Salah Uddin Yusuf
Md. Shakib Hasan Rudro
Professor,
Roll: 1803085
Department of EEE,
Department of EEE,
Khulna University of Engineering & Technology Khulna University of Engineering & Technology
DECLARATION
This is to certify that the thesis work, An improved deep learning based approach of
steering angle prediction for autonomous vehicle by Md. Shakib Hasan Rudro was
performed under the supervision of Prof. Dr. Md. Salah Uddin Yusuf in the Department of
Electrical Electronic Engineering, Khulna University of Engineering & Technology,
Khulna, Bangladesh. The above thesis work has not been submitted anywhere for any
degree.
The above claims are accurate. This study was submitted as an undergraduate thesis.
Signature of Supervisor
Signature of Student
ACKNOWLEDGMENT
First of all, I express my sincere gratitude to the Almighty for His blessings, which have
facilitated the successful completion of this significant academic endeavor.
I extend my deepest appreciation to my honorable supervisor, Prof. Dr. Md. Salah Uddin
Yusuf, Head of the Department of Electrical and Electronic Engineering, Khulna University
of Engineering & Technology, for his invaluable guidance and unwavering support
throughout the course of this thesis. His insightful advice, encouragement, and constant
supervision have been instrumental in shaping the direction of this research.
I am also grateful to all the honorable faculty members of the Department of Electrical and
Electronic Engineering for their cooperation, encouragement, and scholarly insights, which
have enriched this work and contributed to its academic rigor.
Furthermore, I would like to extend my heartfelt thanks to my family and friends for their
unwavering support, encouragement, and understanding during this academic journey.
February 2024
Author
Md. Shakib Hasan Rudro
i
ABSTRACT
This research presents an improved deep learning approach for steering angle prediction in
autonomous vehicles. The study focuses on enhancing the accuracy and robustness of
steering angle prediction models, which are crucial for safe and reliable autonomous driving
systems. Leveraging advanced deep learning techniques, including convolutional neural
networks (CNNs). the proposed approach integrates image data from onboard cameras with
vehicle sensor data to predict steering angles in real-time. The research explores novel
architectures, data pre-processing methods, and training strategies to optimize model
performance and generalization capabilities. Experimental evaluations conducted on realworld driving datasets demonstrate the effectiveness and efficiency of the proposed approach
compared to existing methods. The findings of this study contribute to the advancement of
autonomous vehicle technology and have significant implications for enhancing road safety
and transportation efficiency.
Keywords: Autonomous vehicle, End to end learning, CNN, Semantic Segmentation
ii
Table of Contents
CHAPTER I: INTRODUCTION
1
1.1. Introduction
1
1.2. Problem Description
2
1.3. Motivation
3
1.4. Objectives
5
1.5. Application
5
CHAPTER II: LITERATURE REVIEW
6
2.1. Literature Review
6
CHAPTER III: TERMINOLOGY
8
3.1. Autonomous vehicle definition
8
3.2. End to end learning
8
3.3. CNN
9
3.4. Segmentation
10
3.4.1. FCN-8
12
3.4.2. UNET
13
3.5. Simulator
14
3.5.1. AirSim
15
3.5.1.1. AirSim Block Diagram
16
3.5.1.2. Why airsim
16
3.5.1.3. Graphical User Interface
17
iii
CHAPTER IV: METHODOLOGY
18
4.1. Block Diagram
18
4.2. Methodology
18
4.2.1. Data acquisition
18
4.2.2. Data pre-processing
18
4.2.2.1. Augmentation
19
4.2.3. Segmentation
21
4.2.4. Model Architectures
22
4.2.4.1. Nvidia pilotnet
22
4.2.4.2. Model A
23
4.2.4.3. Model B
24
4.2.5. Training and testing
25
4.2.6. Loss function
25
4.2.7. Evaluation
25
4.3. Challenges
26
CHAPTER V: RESULTS AND DISCUSSION
5.1. Result and discussion
28
28
5.1.1. The Actual Vs predicted steering angle
28
5.1.2. Time comparison
30
5.1.3. Autonomy Comparison
31
5.1.4. FPS vs Maximum Speed
33
5.1.5. Number of data vs Autonomy
34
iv
CHAPTER VI: CONCLUSION
35
6.1. Conclusion
35
6.2. Future work
36
References
37
v
Table of Figures
Figure 1: An autonnomous vehicle with different sensors and camera
8
Figure 2: Traditional vs End to end approach
9
Figure 3: Basic CNN architecture
10
Figure 4: Road Segmentation
11
Figure 5: FCN8 architecture
12
Figure 6: UNET architecture
13
Figure 7: Preview of the airsim simulator
15
Figure 8: Airsim block diagram
16
Figure 9: Graphical user interface
17
Figure 10: Block diagram of the experiment
18
Figure 11: Input image after crop and resize
19
Figure 12: Random brightness
19
Figure 13: Random shifft
20
Figure 14: Random shadow
20
Figure 15: Random flip
21
Figure 16: FCN8 architecture
21
Figure 17: Nvidia pilotnet architecture
22
Figure 18: Proposed model A
23
Figure 19: Proposed model B
24
vi
Figure 20: Challenges of autonomous vehicle
27
Figure 21: Steering angle (actual vs prediction) for model A
28
Figure 22: Steering angle (actual vs prediction) for model B
29
Figure 23: FPS comparison of the baseline model and model A and B
30
Figure 24: Autonomy comparison between various model for (1.5) hours time span for road
with obstacle
32
Figure 25: Fps vs maximum speed
33
vii
Index of Tables
Table 1: Hardware specification
30
viii
(1) CHAPTER I: INTRODUCTION
1.1. Introduction
The emergence of Deep Convolutional Neural Networks (CNNs) has ushered in a
transformative era in the field of autonomous driving. With the fusion of cutting-edge
computer vision, sensor technology, and artificial intelligence, deep CNN-based selfdriving cars have made remarkable strides in recent years. [1] These vehicles represent
the pinnacle of autonomous mobility, offering the promise of safer, more efficient, and
sustainable transportation solutions.
The core premise of deep CNN-based self-driving car research is to replicate and enhance
the human ability to perceive and navigate the complex dynamics of the road
environment. [2] These vehicles leverage deep neural networks to interpret vast streams
of sensor data, including images, LiDAR, radar, and GPS information, in real-time. By
mimicking human visual perception and cognitive decision-making processes, these
intelligent machines can detect obstacles, interpret road signs, and make split-second
decisions to ensure the safety of passengers, pedestrians, and other road users.
An autonomous vehicle is the type of vehicles that has the ability to operate itself as well
as perform required functions without the necessity of any kind of human intervention. It
also capable of sensing its surroundings and take decision from it. Autonomous vehicles
has the potential to transform the way we travel and commute. A successful development
of an autonomous vehicle requires precise testing and evaluation.
Simulation is a critical tool for testing the behavior of autonomous vehicles in various
traffic scenarios. The use of neural networks has shown significant promise in accurately
modeling traffic dynamics. [3]
1
While the concept of self-driving cars holds great promise, it is not without its challenges.
Driving is an inherently complex task, and the intricacies of road regulations and
unpredictable scenarios make navigating busy roads a daunting challenge, even for
human drivers. [4] Therefore, achieving fully autonomous vehicles cannot rely solely on a
single deep learning model. Instead, it necessitates the implementation of a sophisticated
combination of separately trained neural network models. In this research landscape, I
delve into the intricacies of self-driving car systems and explore the challenges and
opportunities presented by these vehicles, ranging from robust perception and sensor
fusion to decision-making algorithms and regulatory considerations. My objective is to
establish a comprehensive pipeline incorporating carefully selected deep learning
techniques that collectively enhance safety, efficiency, and accessibility in transportation.
This vision aims to create a future where autonomous vehicles seamlessly coexist with
traditional human-driven cars. [5]
This report is to investigate the use of neural networks and to simulate the behavior of
autonomous vehicles. The primary objective is the development and implementation of a
neural network-based model. It will be used to test the behavior of an autonomous vehicle
in complex traffic scenarios. I will investigate the performance of the neural network
model in various scenarios. The results of this study will provide valuable insights into
the effectiveness of using neural networks for autonomous vehicles. The proposed study
has the potential to contribute to the development of safe and efficient autonomous
vehicle technologies— which can ultimately benefit society by reducing traffic accidents,
improving transportation efficiency reducing carbon emissions. [6]
1.2. Problem Description
Conventional autonomous driving systems adopt a modular deployment strategy, wherein
each functionality, such as perception, prediction, and planning, is individually developed
and integrated into the onboard vehicle. The planning or control module, responsible for
generating steering and acceleration outputs, plays a crucial role in determining the
driving experience. The most common approach for planning in modular pipelines
involves using sophisticated rule-based designs, which are often ineffective in addressing
2
the vast number of situations that occur while driving. Therefore, there is a growing trend
to leverage large-scale data and to use learning-based planning as a viable alternative.
We define end-to-end autonomous driving systems as fully differentiable programs that
take raw camera data as input and produce control actions as output.
1.3. Motivation
In the classical pipeline, each model serves a standalone component and corresponds to a
specific task (e.g., lane detection). Such a design is beneficial in terms of interpretability,
verifiability, and ease of debugging. [7] However, since the optimization goal across
modules is different, with detection in perception pursuing mean average precision (mAP)
while planning aiming for driving safety and comfort, the entire system may not be
aligned with a unified target. Errors from each module, as the sequential procedure
proceeds, could be compounded and result in an information loss for the driving system.
Moreover, the multi-task, multi-model deployment may increase the computational
burden and potentially lead to sub-optimal use of computation.
In contrast to its classical counterpart, an end-to-end autonomous system offers several
advantages. [8]
(a) The most apparent merit is its simplicity in combining perception, prediction, and
planning into a single model that can be jointly trained.
(b) The whole system, including its inter- mediate representations, is optimized
towards the ultimate task.
(c) Shared backbones increase computational efficiency.
(d) Data-driven optimization has the potential to offer emergent abilities that improve
the system by simply scaling training resources.
The development of self-driving cars represents one of the most transformative and
promising advancements in the field of transportation. Autonomous vehicles have the
potential to revolutionize our daily lives, making transportation safer, more efficient, and
environmentally friendly. [4] At the heart of this groundbreaking technology lies
computer vision, a field that has seen unprecedented growth and innovation in recent
3
years. This paper is motivated by the profound impact that computer vision techniques are
having on the realization of autonomous driving and the myriad challenges and
opportunities they present.
1. Safety and Efficiency: Human error is a leading cause of traffic accidents
worldwide. Self-driving cars, empowered by computer vision, offer the promise of
significantly reducing accidents by providing vehicles with the ability to perceive
their surroundings, make decisions, and execute maneuvers with precision. By
eliminating the risk associated with distracted or impaired driving, self-driving
cars have the potential to save countless lives and reduce the economic toll of
accidents.
2. Accessibility and Mobility: Autonomous vehicles have the potential to
revolutionize transportation for individuals with disabilities and the elderly.
3. Environmental Impact: Autonomous driving can contribute to a more sustainable
future by optimizing traffic flow, reducing congestion, and minimizing fuel
consumption.
4. Technological Advancements: The rapid evolution of computer vision techniques,
driven by deep learning and artificial intelligence, has opened new possibilities for
autonomous vehicles. My goal is to explore the latest developments in computer
vision and their application to self-driving cars, shedding light on cutting-edge
research and technological innovations.
5. Challenges and Ethical Considerations: While the potential benefits of selfdriving cars are vast, there are numerous challenges to overcome, including
ethical dilemmas, regulatory frameworks, and cybersecurity concerns.
4
1.4. Objectives
The objectives of the simulation of autonomous vehicle by using neural network are:
1. To develop an optimized deep CNN based model to predict
steering angle for
self driving car.
2. To evaluate the models and compare them with the Nvidia pilotnet model.
3. To visualize the performance of the models using airsim simulator.
1.5. Application
Application of autonomous vehicle are the following:
1. Enhanced public transportation systems.
2. Improved ride-sharing services.
3. Increased accessibility for all.
4. Efficient emergency response.
5. Streamlined last-mile delivery.
6. Automation in agriculture.
7. Military reconnaissance and logistics.
8. Impact on urban planning and infrastructure.
5
(2) CHAPTER II: LITERATURE REVIEW
2.1. Literature Review
Torabi et al. [9] introduced the term behavior cloning. The process of reconstructing the
human sub cognitive skill through the computer program is referred to as behavioral
cloning. Here, actions of human performing the skill are recorded along with the situation
that gave rise to the action. Human skills such as driving can be reconstructed through
recorded actions which are maintained in a structured way by using learning algorithms in
terms of various manifestation traces to reproduce the skilled behavior.
Bojarski et al. [10] started their research work at NVIDIA on self-driving car inspired by
the ALVINN and DARPA projects. The motivation for their work was to create an end-toend model which enables steering of the car without manual intervention [1] while
recording humans driving behavior along with the steering angle at every second.
Based on the NVIDIA proposed architecture pilotnet (as shown in Fig. 2), Viswanath et
al. from Texas Instruments released JacintoNet i.e an end-to-end neural network for
embedded system vehicles such as tiny humanoid robots.
Xu et al. [11] trained a neural network for predicting discrete or continuous actions also
based on camera inputs. Codevilla et al. also trained a network using camera inputs and
conditioned on high-level commands to output steering and acceleration. This is the first
model that doesn’t just follow lane but also incorporates high level commands. They also
evaluated their approach in realistic simulations of urban driving and on a 1/5 scale
robotic truck.
Kuefler et al. [12] use Generative Adversarial Imitation Learning (GAIL) with simple
affordance- style features as inputs to overcome cascading errors typically present in
behavior cloned policies so that they are more robust to perturbations.
Hecker et al. [13] used 360-degree camera inputs instead of a single front facing camera
and desired route planner to predict steering and speed.
6
M¨uller et al. [5] train a system in simulation using CARLA by training a driving policy
from a scene segmentation network to output high-level control, thereby enabling transfer
learning to the real world using a different segmentation network trained on real data.
Bansal et al. [14] in a research at Waymo (Google) presents another model called
ChauffeurNet, which then outputs a driving trajectory that is consumed by a controller
which translates it to steering and acceleration.
Saifullah et al. [15] used a transformer-based approach for behavior cloning. We’ve
already seen the effectiveness of transformer model in natural language processing
engines such as ChatGPT. The authors proposed a model called BeT (Behavior
Transformer).they claim that this model harnessing the power of transformer can handle
complex tasks in a humanly manner.
7
(3) CHAPTER III: TERMINOLOGY
3.1. Autonomous vehicle definition
Autonomous vehicles, also known as self-driving cars or driverless cars, are vehicles
equipped with advanced technologies that enable them to navigate and operate without
human intervention. These vehicles utilize a combination of sensors, cameras, radar, lidar,
GPS, and sophisticated algorithms to perceive their surroundings, interpret sensory
inputs, and make decisions to navigate safely to their destination. [16]
Figure 1: An autonomous vehicle with different sensors and
camera
Autonomous vehicles have the potential to revolutionize transportation by offering
numerous benefits, including increased road safety, reduced traffic congestion, improved
energy efficiency, and enhanced mobility for individuals with disabilities or limited
access to transportation. They hold promise for transforming various industries, including
transportation, logistics, and urban planning.
3.2. End to end learning
In end-to-end learning, the neural network is trained using a large dataset of input-output
pairs, where the inputs are raw sensor data collected from the vehicle's environment, and
the outputs are corresponding control commands executed by the vehicle's actuators. The
network learns to extract relevant features from the raw input data and generate
8
appropriate control commands without explicit feature engineering or manual rule-based
programming. [17]
Figure 2: Traditional vs End to end approach
End-to-end learning offers several potential advantages for autonomous vehicles,
including simplicity, scalability, and the ability to adapt to diverse driving conditions. By
directly learning driving policies from data, end-to-end approaches have the potential to
capture complex driving behaviors and adapt to novel situations that may not have been
explicitly programmed.
3.3. CNN
Convolutional Neural Networks (CNNs) are a class of deep neural networks that have
revolutionized the field of computer vision. Inspired by the organization of the visual
cortex in animals, CNNs are particularly well-suited for tasks such as image recognition,
object detection, and image classification.
CNNs consist of multiple layers, including convolutional layers, pooling layers, and fully
connected layers. The convolutional layers apply a set of learnable filters to the input
image, detecting features such as edges, textures, and shapes. Pooling layers reduce the
spatial dimensions of the feature maps, while preserving important information. Fully
connected layers combine the features learned by the previous layers to make predictions
about the input data. [18]
9
One of the key advantages of CNNs is their ability to automatically learn hierarchical
representations of features directly from raw input data. This hierarchical feature learning
enables CNNs to achieve superior performance on a wide range of visual recognition
tasks compared to traditional machine learning algorithms.
CNNs have been widely adopted in various applications, including image classification,
object detection, facial recognition, medical image analysis, and autonomous vehicles.
Their success can be attributed to their ability to automatically learn relevant features
from large amounts of data, their scalability to handle complex and high-dimensional
inputs, and their ability to generalize well to new, unseen data.
Figure 3: Basic CNN architecture
3.4. Segmentation
In the context of autonomous vehicles, image segmentation plays a crucial role in
understanding the vehicle's surroundings and making informed decisions for navigation
and control. [19] Here's how segmentation is utilized:
10
Figure 4: Road Segmentation
1. Road and Lane Segmentation: Segmenting the road and lane markings from the
surrounding environment is essential for autonomous vehicles to navigate safely
within lanes. This allows the vehicle to stay centered in its lane and make
appropriate adjustments for turns, lane changes, and other maneuvers.
2. Obstacle Detection and Segmentation: Segmentation techniques are used to
identify and delineate obstacles such as vehicles, pedestrians, cyclists, and other
objects in the vehicle's path. By segmenting obstacles from the background,
autonomous vehicles can assess potential collision risks and plan avoidance
strategies accordingly.
3. Semantic Understanding of the Environment: Semantic segmentation provides
a detailed understanding of the scene by segmenting different elements such as
roads, sidewalks, buildings, vegetation, and other infrastructure. This semantic
understanding enhances the vehicle's situational awareness and enables more
informed decision-making.
4. Localization and Mapping: Segmentation aids in generating high-definition
maps of the environment by accurately delineating various features and
11
landmarks. These maps, often referred to as semantic maps, facilitate precise
localization and navigation of autonomous vehicles by providing detailed
information about the surroundings.
Overall, image segmentation is a critical component of perception systems in autonomous
vehicles, enabling them to interpret and understand the visual information from onboard
sensors and navigate safely and effectively in complex real-world environments.
3.4.1. FCN-8
FCN-8, introduced by Long et al., is renowned for its capability to perform end-to-end
pixel-wise classification. It replaces the fully connected layers of traditional CNNs with
convolutional layers, enabling it to accept inputs of arbitrary sizes and produce dense
predictions. FCN-8 utilizes skip connections from earlier layers to preserve spatial
information, which helps in generating detailed segmentation maps. Additionally, it
incorporates transposed convolutions to upsample feature maps and restore the resolution
of the output. While FCN-8 is efficient and flexible, it may struggle with capturing fine
details due to downsampling operations.
Figure 5: FCN8 architecture
12
3.4.2. UNET
On the other hand, U-Net, proposed by Ronneberger et al., is characterized by its
symmetric encoder-decoder architecture with skip connections. The contracting path of
U-Net resembles a typical CNN encoder, gradually reducing spatial dimensions and
extracting high-level features. However, unlike FCN-8, U-Net's expansive path employs
transposed convolutions for upsampling while also integrating skip connections from the
contracting path. These skip connections facilitate the fusion of low-level and high-level
features, aiding in precise localization and segmentation of objects. U-Net's architecture is
particularly advantageous for medical image segmentation tasks and scenarios where
detailed localization is crucial. Despite its effectiveness, U-Net may be computationally
expensive due to the large number of parameters, especially in deeper architectures.
Figure 6: UNET architecture
13
3.5. Simulator
Simulators play a crucial role in various fields, including autonomous vehicles, aerospace,
robotics, and healthcare, for several reasons:
1. Cost-Effective Testing: Simulators provide a cost-effective alternative to realworld testing. They allow researchers and developers to conduct extensive testing
and validation of systems and algorithms without the need for expensive physical
prototypes or equipment. This significantly reduces development costs and risks
associated with real-world testing.
2. Safety: Simulators offer a safe and controlled environment for testing complex
systems and algorithms, especially those designed for critical applications such as
autonomous vehicles and medical devices. Simulated environments enable
researchers to identify and address potential safety issues without putting human
lives or valuable resources at risk.
3. Re-producibility: Simulators enable researchers to reproduce and control specific
scenarios and conditions with precision. This reproducibility ensures consistency
in testing procedures and results, making it easier to evaluate and compare
different approaches, algorithms, and systems.
4. Scalability: Simulators allow researchers to scale experiments and simulations to
a larger scope and complexity than would be feasible in the real world. This
scalability enables the evaluation of systems and algorithms under a wide range of
scenarios, including rare or extreme conditions that may be difficult to encounter
in reality.
5. Accessibility: Simulators make experimental setups and testing environments
accessible to a broader audience of researchers, developers, and enthusiasts. They
democratize access to advanced technologies and enable collaboration and
knowledge sharing across disciplines and geographic locations. [20]
14
In summary, simulators are necessary tools for research, development, and testing in
various fields due to their cost-effectiveness, safety, reproducibility, scalability, iterative
development capabilities, and accessibility. They enable researchers and developers to
accelerate innovation, mitigate risks, and advance the state-of-the-art in their respective
domains.
3.5.1. AirSim
Figure 7: Preview of the airsim simulator
Microsoft AirSim is an open-source, cross-platform simulator for autonomous vehicles
(AVs), robotics research, and artificial intelligence (AI) development. Developed by
Microsoft Research, AirSim provides a realistic simulation environment for training and
testing autonomous systems in various scenarios without the need for physical prototypes
or real-world testing.
15
3.5.1.1. AirSim Block Diagram
Image
Interpreter
AirSim
Visual Output
Steering command
Figure 8: Airsim block diagram
The communication between the interpreter and airsim happens via TCP port. Airsim
sends a stream of image of desired dimension to the interpreter and the interpreter
provides steering commands after processing the images and sends them back to the
simulator. The simulator then shows a video of the performance of the vehicle based on
the provided command. Autonomy is computed by observing the output of Airsim.
3.5.1.2. Why airsim
Key features of Microsoft AirSim include:
1. High-Fidelity Simulation: AirSim offers a high-fidelity physics engine and
realistic graphics, enabling researchers and developers to create immersive and
realistic simulation environments for AVs and robotics applications.
2. Support for Multiple Platforms: AirSim is designed to be compatible with
multiple platforms, including Windows, Linux, and macOS, making it accessible
to a wide range of developers and researchers.
3. Extensibility and Customization: AirSim provides a modular architecture that
allows users to extend and customize the simulator to suit their specific research
needs. Users can integrate custom vehicle models, sensors, environments, and
algorithms into AirSim for experimentation and testing.
4. Built-in Sensor Simulation: AirSim supports various sensors commonly used in
AVs and robotics, including cameras, lidar, radar, and GPS. These sensors can
generate realistic data streams that mimic real-world sensor outputs, enabling
researchers to develop and validate perception and control algorithms in a
simulated environment.
16
5. Integration with AI Frameworks: AirSim seamlessly integrates with popular AI
frameworks such as TensorFlow and PyTorch, allowing researchers to leverage
state-of-the-art machine learning algorithms for perception, planning, and control
tasks within the simulator.
Overall, Microsoft AirSim provides a powerful and flexible platform for researchers and
developers to accelerate innovation in autonomous systems by enabling rapid
prototyping, testing, and validation in a simulated environment. Its realistic simulation
capabilities and extensibility make it an invaluable tool for advancing the field of AVs and
robotics. [21]
3.5.1.3. Graphical User Interface
Figure 9: Graphical user interface
A graphical user interface was designed to visualize the steering angle and also the speed
curve of the vehicle. It also shows the FPS and numerical value of the steering angle. This
can be operated in both manual and autonomous mode. In the manual mode, it works as a
data collector. Using this Graphical User Interface(GUI) autonomy and the maximum
speed for a certain FPS can be calculated.
17
(4) CHAPTER IV: METHODOLOGY
4.1. Block Diagram
Figure 10: Block diagram of the experiment
4.2. Methodology
4.2.1. Data acquisition
Utilizing the AirSim simulator, I conducted data collection procedures to assemble
comprehensive datasets for training and validating autonomous vehicle algorithms. This
involved capturing images from multiple viewpoints, including the center, left, and right
perspectives, along with recording the corresponding steering angles. By incorporating
data from various viewpoints, this approach aims to enhance the robustness and
generalization capabilities of the autonomous vehicle system, thereby facilitating more
accurate and reliable navigation in diverse real-world scenarios.
4.2.2. Data pre-processing
In the data pre-processing phase, meticulous attention was given to maintaining
uniformity across all collected images. Each image underwent cropping and resizing to
18
Figure 11: Input image after crop and resize
standard dimensions of 66×200 pixels, ensuring consistency in the dataset. Subsequently,
three types of augmentation techniques were applied: random brightness adjustments,
random shifts, and random flips. These augmentation strategies were implemented to
enhance the robustness and variability of the dataset, thereby facilitating more effective
training of the autonomous vehicle model. [23] Concurrently, the corresponding steering
angles for each augmented image were meticulously logged in a CSV file, ensuring
precise alignment between the image data and steering control inputs. This systematic
pre-processing methodology ensured the preparation of a high-quality dataset conducive
to the training process.
4.2.2.1. Augmentation
Random Brightness:
Figure 12: Random brightness
19
Random shift:
Figure 13: Random shifft
Random shadow:
Figure 14: Random shadow
20
Random flip:
Figure 15: Random flip
4.2.3. Segmentation
Semantic segmentation, achieved through techniques like Fully Convolutional Networks
(FCNs), partitions an image into meaningful regions, enhancing scene understanding and
object localization which improved perception capabilities, robustness to environmental
variations, and support for end-to-end learning. FCN8, chosen for its speed, enables realtime processing which is crucial for autonomous driving.
Figure 16: FCN8 architecture
While UNet offers superior performance, FCN8's efficiency ensures the model operates
swiftly, crucial for timely decision-making in dynamic environments. Thus, FCN8
balances performance and speed, ideal for real-time applications like autonomous driving.
These advancements contribute to the development of more effective and reliable
21
autonomous driving systems capable of navigating safely in complex real-world
environments. [24]
4.2.4. Model Architectures
4.2.4.1. Nvidia pilotnet
Figure 17: Nvidia pilotnet architecture
22
4.2.4.2. Model A
Figure 18: Proposed model A
23
4.2.4.3. Model B
Figure 19: Proposed model B
24
4.2.5. Training and testing
The training process involves preparing and augmenting datasets, training the model
using
optimization
algorithms,
and
fine-tuning
hyper-parameters
for
optimal
performance. Validation assesses the model's performance on a separate dataset, while
testing evaluates its generalization to real-world scenarios. Finally, deployment in the
target environment requires continuous monitoring and updating to maintain
effectiveness.
4.2.6. Loss function
Mean Square Error (MSE) is a common loss function used in regression tasks, including
machine learning and neural network training. It measures the average squared difference
between the predicted values and the actual values in a dataset. [12]
In the context of training models for tasks such as regression, MSE calculates the average
squared difference between the predicted output and the ground truth labels. It penalizes
larger errors more heavily than smaller ones, making it suitable for tasks where precise
prediction accuracy is important.
Mathematically, MSE is calculated by taking the average of the squared differences
between predicted and actual values:
n
MSE=
1
∑ ( y − y ' )2
n i=1 i i
where n is the number of samples, yi represents the actual value, and y’i represents the
predicted value.
Overall, MSE is a useful loss function for training regression models, providing a
quantitative measure of the model's performance by quantifying the average squared
difference between predicted and actual values.
4.2.7. Evaluation
The percentage of the time the network could drive the car without human intervention is
defined as autonomy. The metric is determined by counting simulated human
interventions. These interventions occur when the simulated vehicle departs from the
25
center line by more than one meter. We assume that in real life an actual intervention
would require a total of six seconds: this is the time required for a human to retake control
of the vehicle, re-center it, and then restart the self-steering mode. We calculate the
percentage autonomy by counting the number of interventions, multiplying by 6 seconds,
dividing by the elapsed time of the simulated test, and then subtracting the result from 1.
[10]
autonomy=(1−
(number of interventions)×6 seconds
)×100
elapsed time [ seconds]
4.3. Challenges
Driving itself is a complex task that takes years to master even for human. So there are
many challenges to overcome in autonomous vehicle implementation. One such challenge
that stands out is the stability issue. If the vehicle somehow drifts from its expected
trajectory, than it gets slightly unfamiliar images as input. Which leads to an error in the
predicted steering angle, which in turn pushes the vehicle toward more unfamiliar region
and the cycle repeats.
As a result the error builds up and the vehicle shows unstable behavior. A visible shaking
can be noticed in such case. In this research, I propose two possible solutions to the
problem.
26
Figure 20: Challenges of autonomous vehicle
1. While training, there was 3 cameras to cover a greater region of the scenario and adjust
them properly along with their corresponding steering angle with respect to the center
image. This step ensures more coverage of the sorrounding.
2. Taking the moving average of the previous few frames with the current frame. This
step minimizes the shaking of the vehicle.
27
(5) CHAPTER V: RESULTS AND DISCUSSION
5.1. Result and discussion
The results of the steering angle prediction model demonstrate its effectiveness in
accurately estimating steering angles based on input images captured from the center, left,
and right viewpoints. Through careful training and validation, the model showcases a
strong ability to infer steering commands, crucial for guiding autonomous vehicles along
desired trajectories.
5.1.1. The Actual Vs predicted steering angle
Figure 21: Steering angle (actual vs prediction) for model A
The graph depicting the comparison between actual and predicted steering angles by
Model A reveals a remarkable resemblance between the two. Notably, the model's
predictions closely mirror the actual steering angles, capturing abrupt changes in direction
almost instantaneously. This close alignment between predicted and actual values
underscores the model's efficacy in accurately estimating steering commands, a critical
aspect for ensuring precise control of autonomous vehicles. The rapid response of the
model to changes in direction further highlights its capability to adapt to dynamic driving
28
scenarios in real-time. Overall, the graph illustrates the impressive performance of Model
A in steering angle prediction, affirming its potential for enhancing the autonomy and
safety of vehicle navigation systems.
Figure 22: Steering angle (actual vs prediction) for model B
The graphical representation of steering angle comparisons between actual and predicted
values for Model B showcases a significant resemblance between the two datasets.
Notably, Model B's predictions closely mirror the actual steering angles, capturing subtle
and abrupt changes in direction with impressive accuracy. This close correspondence
indicates the model's proficiency in accurately estimating steering commands, essential
for ensuring precise vehicle control in diverse driving conditions. [25] Moreover, the
swift response of Model B to changes in direction underscores its adaptability and
responsiveness, further enhancing its suitability for real-world applications.
29
5.1.2. Time comparison
The graph depicting frames per second (FPS) for three different models—Nvidia, Model
A, and Model B—holds significant importance in the context of autonomous vehicles.
Real-time processing is crucial for autonomous vehicles to make rapid decisions and
navigate safely in dynamic environments.
FPS comparison
16
14
Frames/Sec
12
10
8
6
4
2
0
Nvidia pilotnet
Model A
Model B
Models
Figure 23: FPS comparison of the baseline model and model A and B
While Nvidia exhibits the highest FPS among the models, both Model A and Model B
demonstrate slightly lower FPS. However, these FPS values remain within a tolerable
range, indicating that they are still capable of processing frames at a sufficiently high rate
for real-time operation.
The slight sacrifice in FPS for Model A and Model B is justified by their ability to
provide greater autonomy and make accurate predictions, such as steering angle
estimation or semantic segmentation. The trade-off between FPS and autonomy is
therefore worth considering, as it ensures the models can effectively analyze and respond
to the surrounding environment in real-time, ultimately enhancing the safety and
efficiency of autonomous vehicle navigation systems.
30
It's important to note that FPS values are heavily dependent on the specifications of the
device used for testing. The measurements presented in the graph were conducted under
the context of specific device specifications, outlined in the provided table.
Table 1: Hardware specification
CPU
GPU
RAM
Intel Core i5 11th Gen 11300H
Nvidia RTX 3050
16 GB
It's worth emphasizing that FPS performance can vary significantly based on factors such
as CPU and GPU processing power, memory capacity, and optimization techniques
employed. Therefore, upgrading the device specifications, such as utilizing a more
powerful CPU or GPU, increasing memory capacity, or optimizing software
configurations, can lead to significant improvements in FPS for all models. [26]
By investing in hardware upgrades or utilizing more advanced computing resources, such
as high-performance GPUs or dedicated processing units, the FPS of all models can be
substantially enhanced. This underscores the importance of considering device
specifications and resource availability when evaluating FPS performance and optimizing
the performance of autonomous vehicle systems.
5.1.3. Autonomy Comparison
In comparing the autonomy between Model A and Model B with the baseline Nvidia
model, significant improvements are evident, despite utilizing the same amount of data.
Both Model A and Model B demonstrate enhanced autonomy, showcasing advancements
in their ability to make independent decisions and navigate complex environments.
31
Autonomy analysis
100
Autonomy(%)
95
90
With obstacle
Without obstacle
85
80
75
Nvidia pilotnet
Model A
Model B
Models
Figure 24: Autonomy comparison between various model for (1.5) hours time span
for road with obstacle
Model A and Model B exhibit superior autonomy compared to the baseline Nvidia model
due to several factors. Firstly, their improved accuracy in tasks such as steering angle
prediction or semantic segmentation enables more precise and reliable decision-making,
leading to smoother and safer navigation. This increased accuracy is attributed to the
utilization of advanced neural network architectures and optimized training
methodologies.
Furthermore, Model A and Model B showcase enhanced adaptability to diverse driving
conditions and scenarios. Through robust training and validation processes, these models
have learned to generalize effectively from the provided dataset, allowing them to
respond appropriately to real-world challenges such as varying road conditions,
unexpected obstacles, and dynamic traffic patterns.
Moreover, the introduction of novel features or techniques in Model A and Model B may
contribute to their improved autonomy. For instance, the incorporation of additional
32
sensor modalities, advanced fusion techniques, or sophisticated planning algorithms may
further enhance their capabilities beyond those of the baseline Nvidia model.
Overall, the comparison highlights the substantial advancements in autonomy achieved
by Model A and Model B, underscoring the continuous evolution and refinement of
autonomous vehicle technology. These improvements signify a promising trajectory
towards the development of highly autonomous vehicles capable of navigating safely and
efficiently in diverse real-world environments.
5.1.4. FPS vs Maximum Speed
As we know there is a direct relationship between the number of frames that can be
processed per second and the maximum speed at which the vehicle can operate. I have
tested the maximum speed achieved in various FPS and plotted them. From the graph is is
clear that there is a linear relationship between the maximum speed and FPS.
FPS vs maximum speed
16
14
Speed(m/s)
12
10
8
6
4
2
0
10
12
14
FPS
Figure 25: Fps vs maximum speed
33
16
5.1.5. Number of data vs Autonomy
All of the models were trained on different amount of data. We can see in the following
chart that the autonomy increased as the amount of data increased. But in the lower
region proposed model A and B shows significantly greater autonomy than the Nvidia
pilotnet model trained on the same amount of data. But as we increase the the size of the
dataset, autonomy of all the models becomes close. This graph shows us that the proposed
models perform better even if trained on low amount of data. Which is necessary because
collecting data for autonomous vehicle is expensive, and risky. Specially for regions like
Bangladesh collecting huge data of various scenario is a challenge. So the proposed
models will perform better in such scenarios.
Autonomy(%)
Autonomy vs the ammount of data
100
90
80
70
60
50
40
30
20
10
0
Nvidia
Model A
Model B
9k
23k
50k
No. of training data
34
92k
(6) CHAPTER VI: CONCLUSION
6.1. Conclusion
The thesis explores the development and enhancement of autonomous vehicle technology,
focusing on key aspects such as perception, decision-making, and autonomy. [27]
Through rigorous experimentation and analysis, several significant findings and
advancements have been achieved, culminating in a comprehensive understanding of
autonomous vehicle systems and their potential impact on transportation and society.
The research begins with an investigation into perception algorithms, particularly
semantic segmentation and steering angle prediction, crucial for understanding the
vehicle's surroundings and making informed decisions. By employing advanced neural
network architectures such as FCN8 and UNet, significant improvements in accuracy and
reliability are achieved, leading to more precise perception capabilities. [28]
Furthermore, the thesis delves into the decision-making process, exploring the intricacies
of steering control and trajectory planning in autonomous vehicles. Through the
development of sophisticated decision-making algorithms and optimization techniques,
the models demonstrate enhanced autonomy and adaptability, capable of navigating
complex environments with confidence and efficiency.
Moreover, the research emphasizes the importance of real-time performance and
computational efficiency in autonomous vehicle systems. [7] By evaluating the frames
per second (FPS) of different models and considering device specifications, valuable
insights are gained into the trade-offs between autonomy and computational resources,
guiding future optimizations and advancements in system design. [13]
Overall, the thesis contributes significant insights and advancements to the field of
autonomous vehicle technology, paving the way for safer, more efficient, and more
autonomous transportation systems. By leveraging cutting-edge technologies and
methodologies, autonomous vehicles hold the potential to revolutionize mobility, reshape
urban landscapes, and enhance the quality of life for people around the world. As we
continue to push the boundaries of innovation and research in this field, the future of
35
autonomous vehicles remains bright, promising a world where transportation is safer,
more sustainable, and more accessible than ever before.
6.2. Future work
The proposed deep learning approach for steering angle prediction in autonomous
vehicles presents a promising foundation for further exploration and development. Here
are some potential avenues for future work:
1. Investigate the impact of deeper and more complex neural network
architectures: Explore deeper convolutional neural networks (CNNs), recurrent
neural networks (RNNs), or a combination of both (CNN-RNN) to capture
complex temporal and spatial dependencies within the sensor data.
2. Incorporate additional sensor modalities: Integrate data from various sensors like
LiDAR, radar, and GPS alongside camera images to provide a more
comprehensive understanding of the environment.
3. Conduct extensive on-road testing: Evaluate the performance of the developed
model in real-world driving scenarios under diverse weather conditions, traffic
patterns, and road infrastructure variations.
4. Develop safety and reliability measures: Implement mechanisms for anomaly
detection, fault tolerance, and explainability to ensure the safe and reliable
operation of the autonomous vehicle in real-world scenarios.
5. Address ethical concerns surrounding autonomous vehicles: Explore the ethical
implications of self-driving cars in terms of decision-making in critical
situations, liability in case of accidents, and potential biases in the training data.
By addressing these future research directions, this can contribute to advancing the stateof-the-art in autonomous vehicle technology, ultimately paving the way towards safer,
more efficient, and more widely adopted autonomous transportation systems.
36
(7) References
[1] S. OwaisAli Chishti, S. Riaz, M. BilalZaib, and M. Nauman, “Self-Driving Cars
Using CNN and Q-Learning,” in 2018 IEEE 21st International Multi-Topic
Conference (INMIC), Nov. 2018, pp. 1–7. doi: 10.1109/INMIC.2018.8595684.
[2] J. del Egio, L. M. Bergasa, E. Romera, C. Gómez Huélamo, J. Araluce, and R. Barea,
“Self-driving a Car in Simulation Through a CNN,” in Advances in Physical Agents,
Springer, Cham, 2019, pp. 31–43. doi: 10.1007/978-3-319-99885-5_3.
[3] A. Bhalla, M. S. Nikhila, and P. Singh, “Simulation of Self-driving Car using Deep
Learning,” in 2020 3rd International Conference on Intelligent Sustainable Systems
(ICISS), Dec. 2020, pp. 519–525. doi: 10.1109/ICISS49785.2020.9315968.
[4] J. Y. C. Chen and J. E. Thropp, “Review of Low Frame Rate Effects on Human
Performance,” IEEE Trans. Syst. Man Cybern. - Part Syst. Hum., vol. 37, no. 6, pp.
1063–1076, Nov. 2007, doi: 10.1109/TSMCA.2007.904779.
[5] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun, “Driving Policy Transfer via
Modularity and Abstraction.” arXiv, Dec. 13, 2018. doi: 10.48550/arXiv.1804.09364.
[6] J. Ni, K. Shen, Y. Chen, W. Cao, and S. X. Yang, “An Improved Deep Network-Based
Scene Classification Method for Self-Driving Cars,” IEEE Trans. Instrum. Meas., vol.
71, pp. 1–14, 2022, doi: 10.1109/TIM.2022.3146923.
[7] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy, “End-to-end
Driving via Conditional Imitation Learning.” arXiv, Mar. 02, 2018. doi:
10.48550/arXiv.1710.02410.
[8] K. Gauen et al., “Comparison of Visual Datasets for Machine Learning,” 2017 IEEE
Int. Conf. Inf. Reuse Integr. IRI, pp. 346–355, Aug. 2017, doi: 10.1109/IRI.2017.59.
37
[9] F. Torabi, G. Warnell, and P. Stone, “Behavioral Cloning from Observation,” pp.
4950–4957, 2018, Accessed: Feb. 21, 2024. [Online]. Available:
https://www.ijcai.org/proceedings/2018/687
[10]
M. Bojarski et al., “End to End Learning for Self-Driving Cars.” arXiv, Apr. 25,
2016. doi: 10.48550/arXiv.1604.07316.
[11]
H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end Learning of Driving Models
from Large-scale Video Datasets.” arXiv, Jul. 23, 2017. doi:
10.48550/arXiv.1612.01079.
[12]
A. Kuefler, J. Morton, T. Wheeler, and M. Kochenderfer, “Imitating Driver
Behavior with Generative Adversarial Networks.” arXiv, Jan. 23, 2017. doi:
10.48550/arXiv.1701.06699.
[13]
S. Hecker, D. Dai, and L. Van Gool, “End-to-End Learning of Driving Models
with Surround-View Cameras and Route Planners.” arXiv, Aug. 06, 2018. doi:
10.48550/arXiv.1803.10158.
[14]
M. Bansal, A. Krizhevsky, and A. Ogale, “ChauffeurNet: Learning to Drive by
Imitating the Best and Synthesizing the Worst.” arXiv, Dec. 07, 2018. doi:
10.48550/arXiv.1812.03079.
[15]
N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto, “Behavior
Transformers: Cloning k modes with one stone.” arXiv, Oct. 11, 2022. doi:
10.48550/arXiv.2206.11251.
[16]
A. Faisal, M. Kamruzzaman, T. Yigitcanlar, and G. Currie, “Understanding
autonomous vehicles: A systematic literature review on capability, impact, planning
and policy,” J. Transp. Land Use, vol. 12, no. 1, pp. 45–72, 2019, Accessed: Feb. 21,
2024. [Online]. Available: https://www.jstor.org/stable/26911258
38
[17]
T. Glasmachers, “Limits of End-to-End Learning,” in Proceedings of the Ninth
Asian Conference on Machine Learning, PMLR, Nov. 2017, pp. 17–32. Accessed:
Feb. 21, 2024. [Online]. Available:
https://proceedings.mlr.press/v77/glasmachers17a.html
[18]
Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A Survey of Convolutional Neural
Networks: Analysis, Applications, and Prospects,” IEEE Trans. Neural Netw. Learn.
Syst., vol. 33, no. 12, pp. 6999–7019, Dec. 2022, doi: 10.1109/TNNLS.2021.3084827.
[19]
Y. Xiao, F. Codevilla, A. Gurram, O. Urfalioglu, and A. M. López, “Multimodal
End-to-End Autonomous Driving,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 1,
pp. 537–547, Jan. 2022, doi: 10.1109/TITS.2020.3013234.
[20]
H. Li, J. Li, X. Guan, B. Liang, Y. Lai, and X. Luo, “Research on Overfitting of
Deep Learning,” in 2019 15th International Conference on Computational
Intelligence and Security (CIS), Dec. 2019, pp. 78–81. doi: 10.1109/CIS.2019.00025.
[21]
S. Shah, D. Dey, C. Lovett, and A. Kapoor, “AirSim: High-Fidelity Visual and
Physical Simulation for Autonomous Vehicles,” in Field and Service Robotics,
Springer, Cham, 2018, pp. 621–635. doi: 10.1007/978-3-319-67361-5_40.
[22]
J. Cui, H. Qiu, D. Chen, P. Stone, and Y. Zhu, “COOPERNAUT: End-to-End
Driving with Cooperative Perception for Networked Vehicles,” in 2022 IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2022, pp.
17231–17241. doi: 10.1109/CVPR52688.2022.01674.
[23]
Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. Van Gool, “End-to-End Urban Driving
by Imitating a Reinforcement Learning Coach.” arXiv, Oct. 04, 2021. doi:
10.48550/arXiv.2108.08265.
[24]
J. Hawke et al., “Urban Driving with Conditional Imitation Learning.” arXiv, Dec.
05, 2019. doi: 10.48550/arXiv.1912.00177.
39
[25]
S. Narayan and G. Tagliarini, “An analysis of underfitting in MLP networks,” in
Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005.,
Jul. 2005, pp. 984–988 vol. 2. doi: 10.1109/IJCNN.2005.1555986.
[26]
R. Chekroun, M. Toromanoff, S. Hornauer, and F. Moutarde, “GRI: General
Reinforced Imitation and its Application to Vision-Based Autonomous Driving.”
arXiv, May 17, 2022. doi: 10.48550/arXiv.2111.08575.
[27]
J. Kolluri, V. K. Kotte, M. S. B. Phridviraj, and S. Razia, “Reducing Overfitting
Problem in Machine Learning Using Novel L1/4 Regularization Method,” in 2020 4th
International Conference on Trends in Electronics and Informatics (ICOEI)(48184),
Jun. 2020, pp. 934–938. doi: 10.1109/ICOEI48184.2020.9142992.
[28]
A. Mikołajczyk and M. Grochowski, “Data augmentation for improving deep
learning in image classification problem,” in 2018 International Interdisciplinary
PhD Workshop (IIPhDW), May 2018, pp. 117–122. doi:
10.1109/IIPHDW.2018.8388338.
40