Reinforcement Learning
Concepts
Sidney Givigi
Course evaluation
Assignment 1
5%
Due Fri, Feb 13 before midnight.
Assignment 2
5%
Due Fri, Mar 20 before midnight.
Quiz 1
25%
Tue, Feb 3 in Chernoff Hall Room 117.
Quiz 2
25%
Tue, Mar 3 in Chernoff Hall Room 117.
Quiz 3
25%
Tue, Mar 24 in Chernoff Hall Room 117.
Project
40%
Due Thu, Apr 2 before midnight.
Reinforcement Learning
• “Reinforcement learning is learning what to do – how to
map situations to actions – so as to maximize a numerical
reward signal. The learner is not told which actions to
take, but instead must discover which actions yield the
most reward by trying them.”
• Sutton and Barto
Reinforcement Learning
• Learning through interaction with environment
• Ideal for games, robots (physical or virtual), vehicles, etc.
– Atari Breakout
– Ms. Pacman
4
Reinforcement Learning
5
Components: Environment
• The environment exists outside the agent.
• The environment is modified by the actions executed by
the agent.
• The agent observes the STATE of the environment (e.g.,
through sensors).
• The environment “generates” rewards based on the
agent’s actions.
Components: Reward
• The reward is received from the environment.
• It is usually designed to incentivize the agent to present a
behaviour.
• Multiple functions may be used for rewards.
Components: Policy
• A policy is the logical structure that generates actions
based on observations
• 𝑎𝑐𝑡𝑖𝑜𝑛𝑠 = 𝑝𝑜𝑙𝑖𝑐𝑦(𝑠𝑡𝑎𝑡𝑒)
• The state representation may be deterministic or not.
• The objective of a reinforcement learning system is to
find a suitable policy for a problem.
Component: Training
• Done by an algorithm.
• Try to find optimal parameters for the policy.
• Try to optimize (maximize or minimize) the rewards.
• Based on the interaction with the environment.
What do we need to know?
• Problem formulation
– How is the problem defined?
– How is the environment represented?
– What are the states?
– What are the observations?
– What are the actions?
• Knowledge representation
– How is policy represented?
– Is the representation discrete or continuous?
– What if we find something we have never seen before?
Example: tic-tac-toe
Example: Game
"#$%&
G()$&*%
!
!*%%+,-%.
!
12
Example: Optimal guard
"#$%&
G()$&*%
!
!*%%+,-%.
I*$01$23*4
I*5+-(
!
!
67,+8$34
!$%5*,
!
13
Example: Irrational guard
"#$%&
G()$&*%
!
!*%%+,-%.
!
!
14
Example: Experiments
15
Questions
16
0
