AIS462 CI
Deep Reinforcement Learning
Lecture 1
Presented By: Ghada Khoriba
Assoc. Prof. of AI
1
Course Learning Outcomes
•
•
•
•
•
•
•
Explains the fundamentals of deep reinforcement learning
Implement Basic Reinforcement Learning Algorithms:
Apply Deep Q-Learning Algorithms
Combine deep Q-learning and policy-gradient methods to
create sophisticated actor-critic algorithms.
Develop Multi-Agent Reinforcement Learning Systems
Apply Attention Mechanisms for Efficient Learning
Explore Advanced Topics and Future Directions in Deep
Reinforcement Learning
2
Course Outline
•
•
•
•
•
•
•
•
•
Reinforcement learning foundations
Modeling reinforcement learning problems
Deep Q-networks
Policy gradient methods
Actor-critic methods
Alternative optimization methods: Evolutionary Algorithms.
Curiosity-driven exploration
Multi-agent reinforcement learning
Interpretable reinforcement learning: Attention and relational models
3
Course extra resources
CS234: Reinforcement Learning Winter 2025
https://web.stanford.edu/class/cs234/index.html
Serrano.Academy
https://www.youtube.com/@SerranoAcademy
Grokking Deep Reinforcement Learning
https://github.com/mimoralea/gdrl/tree/master/notebooks
OpenAIGYM
https://spinningup.openai.com/en/latest/user/installation.html
4
Course Grading Criteria
This course is Project-Based Learning.
Grading items:
❑ Classwork
❑15% Lab Tasks
❑10% Lecture Short Quizzes
❑15% Midterm Exam
❑Final assessment
❑ 30% Course Project
❑30% Final Exam
5
Course Project details
• (27-30 points): Outstanding project, wellimplemented, analyzed, and presented.
5 Points- Problem Definition & Background
• (24-26 points): Good project with solid
5 Points- Algorithm Selection & Justification
implementation but minor weaknesses.
5 Points- Implementation & Code Quality
• (20-23 points): Functional project but lacks
5 Points- Training & Performance Analysis
depth in analysis or presentation.
5 Points- Experimental Results & Visualization
• (15-19 points): Basic project with missing
5 Points- Report & Presentation
components or weak execution.
• (15 points or less): Incomplete or nonfunctional project.
6
What is reinforcement learning?
• Deep reinforcement learning (DRL)
• Dynamic programming versus Monte Carlo
• The reinforcement learning framework
• What can we do with reinforcement learning?
• Why deep reinforcement learning?
7
Deep reinforcement learning (DRL)
Deep reinforcement learning (DRL) is a
machine learning approach to artificial
intelligence concerned with creating
computer programs that can solve
problems requiring intelligence.
The distinct property of DRL programs is
learning through trial and error from
feedback that’s simultaneously sequential,
evaluative, and sampled by leveraging
powerful
non-linear
function
approximation.
8
9
10
Unlike an image classifier, a reinforcement
learning algorithm dynamically interacts with data.
It continually consumes data and decides what actions to take—actions that will
change the subsequent data presented to it. A video game screen might be input data
for an RL algorithm, which then decides which action to take using the game
controller, and this causes the game to update (e.g. the player moves or fires a
weapon).
11
Deep reinforcement learning (DRL)
Reinforcement learning is a generic framework for representing and solving
control tasks, but within this framework, we are free to choose which
algorithms we want to apply to a particular control task.
Deep learning algorithms are a natural choice as they can process complex
data efficiently, and this is why we’ll focus on deep reinforcement learning.
12
Deep reinforcement learning (DRL)
One added complexity of moving from image processing to the
domain of control tasks is the additional element of time.
In control tasks, we similarly have a space
of data to process, but each piece of data
also has a time dimension—the data exists
in both time and space.
In the RL framework, decides which actions to take for
a control task (e.g., driving a robot vacuum), and the
action results in a positive or negative reward, which
will positively or negatively reinforce that action and
hence train the learning algorithm.
13
14
Exploration is any action that lets the
agent discover new features about the
environment, while exploitation is
capitalizing on knowledge already
gained. If the agent continues to
exploit only past experiences, it is
likely to get stuck in a suboptimal
policy.
The agent can approximate
functions using a variety of
ML methods and
techniques, from decision
trees to SVMs to neural
networks.
15
Mathematics allows us to make precise statements
about what’s true and how things are related, and it
offers rigorous explanations for how and why things
work.
Alexander Zai
Brandon Brown
We could teach RL without any math and use
Python, but that approach would handicap you in
understanding future advances.
16