Chapter 7: Eligibility Traces
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
1
Midterm
Mean = 77.33 Median = 82
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
2
N-step TD Prediction
❐ Idea: Look farther into the future when you do TD backup
(1, 2, 3, …, n steps)
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
3
Mathematics of N-step TD Prediction
❐ Monte Carlo:
Rt = rt +1 + "rt + 2 + " 2 rt +3 + L + " T !t !1rT
❐ TD: Rt(1) = rt +1 + !Vt ( st +1 )
Use V to estimate remaining return
❐ n-step TD:
2 step return:
n-step return:
Rt( 2 ) = rt +1 + !rt + 2 + ! 2Vt ( st + 2 )
Rt( n ) = rt +1 + "rt + 2 + " 2 rt +3 + L + " n !1rt + n + " nVt ( st + n )
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
4
Learning with N-step Backups
❐ Backup (on-line or off-line):
!Vt (st ) = " [ Rt( n) # Vt (st )]
❐ Error reduction property of n-step returns
max E! {Rtn | st = s} # V ! ( s ) $ " n max V ( s ) # V ! ( s )
s
s
n step return
Maximum error using n-step return
Maximum error using V
❐ Using this, you can show that n-step methods converge
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
5
Random Walk Examples
❐ How does 2-step TD work here?
❐ How about 3-step TD?
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
6
A Larger Example
❐ Task: 19 state
random walk
❐ Do you think there
is an optimal n (for
everything)?
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
7
Averaging N-step Returns
❐ n-step methods were introduced to help with
TD(λ) understanding
❐ Idea: backup an average of several returns
e.g. backup half of 2-step and half of 4step
Rtavg =
One backup
1 ( 2) 1 ( 4)
Rt + Rt
2
2
❐ Called a complex backup
Draw each component
Label with the weights for that
component
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
8
Forward View of TD(λ)
❐ TD(λ) is a method for
averaging all n-step backups
n-1 (time since
weight by λ
visitation)
λ-return:
#
Rt = (1" ! )$ !n "1Rt(n)
!
n=1
❐ Backup using λ-return:
[
!Vt (st ) = " Rt# $ Vt (st )
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
]
9
λ-return Weighting Function
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
10
Relation to TD(0) and MC
❐ λ-return can be rewritten as:
T " t "1
Rt = (1" ! ) # !n"1 Rt(n) + !T "t "1Rt
!
n=1
Until termination
After termination
❐ If λ = 1, you get MC:
T "t "1
Rt = (1" 1) #1n"1 Rt(n ) + 1T " t "1 Rt = Rt
!
n=1
❐ If λ = 0, you get TD(0)
T "t "1
Rt = (1" 0) # 0 n"1 Rt(n ) + 0T " t "1 Rt = Rt(1)
!
n=1
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
11
Forward View of TD(λ) II
❐ Look forward from each state to determine update from
future states and rewards:
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
12
λ-return on the Random Walk
❐ Same 19 state random walk as before
❐ Why do you think intermediate values of λ are best?
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
13
Backward View of TD(λ)
❐ The forward view was for theory
❐ The backward view is for mechanism
et (s ) " ! +
❐ New variable called eligibility trace
On each step, decay all traces by γλ and increment the
trace for the current state by 1
Accumulating trace
if s $ st
% !"et #1(s)
et (s) = &
'!"et #1(s) + 1 if s = st
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
14
On-line Tabular TD(λ)
Initialize V(s) arbitrarily and e(s) = 0, for all s !S
Repeat (for each episode) :
Initialize s
Repeat (for each step of episode) :
a " action given by # for s
Take action a, observe reward, r, and next state s$
% " r + &V( s$) ' V (s)
e(s) " e(s) + 1
For all s :
V(s) " V(s) + (%e(s)
e(s) " &)e(s)
s " s$
Until s is terminal
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
15
Backward View
# t = rt +1 + "Vt ( st +1 ) ! Vt ( st )
❐ Shout δt backwards over time
❐ The strength of your voice decreases with temporal
distance by γλ
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
16
Relation of Backwards View to MC & TD(0)
❐ Using update rule:
#Vt ( s ) = !" t et ( s )
❐ As before, if you set λ to 0, you get to TD(0)
❐ If you set λ to 1, you get MC but in a better way
Can apply TD(1) to continuing tasks
Works incrementally and on-line (instead of waiting to
the end of the episode)
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
17
Forward View = Backward View
❐ The forward (theoretical) view of TD(λ) is equivalent to
the backward (mechanistic) view for off-line updating
❐ The book shows:
T "1
TD
T "1
$
!V
(s)
=
!V
# t
# t (st )Isst
t=0
t=0
Backward updates Forward updates
algebra shown in book
T "1
TD
T "1
# !Vt (s) = # $ Isst
t=0
t=0
T "1
k"t
#(%& ) ' k
k =t
T #1
T #1
T #1
t=0
k =t
k#t
!V
(s
)I
=
%
I
(
&"
)
'k
$ t t sst $ sst $
"
t=0
❐ On-line updating with small α is similar
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
18
On-line versus Off-line on Random Walk
❐ Same 19 state random walk
❐ On-line performs better over a broader range of parameters
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
19
Control: Sarsa(λ)
❐ Save eligibility for state-action
pairs instead of just states
$!"e (s, a) + 1 if s = st and a = at
et (s, a) = % t #1
otherwise
& !"et #1(s,a)
Qt +1(s, a) = Qt (s, a) + '(t et (s, a)
(t = rt +1 + !Qt (st +1,at +1) # Qt (st , at )
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
20
Sarsa(λ) Algorithm
Initialize Q(s,a) arbitrarily and e(s, a) = 0, for all s, a
Repeat (for each episode) :
Initialize s, a
Repeat (for each step of episode) :
Take action a, observe r, s!
Choose a ! from s! using policy derived from Q (e.g. ? - greedy)
" # r + $Q(s!, a !) % Q(s, a)
e(s,a) # e(s,a) + 1
For all s,a :
Q(s, a) # Q(s, a) + &"e(s, a)
e(s, a) # $'e(s, a)
s # s!;a # a !
Until s is terminal
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
21
Sarsa(λ) Gridworld Example
❐ With one trial, the agent has much more information about how to get
to the goal
not necessarily the best way
❐ Can considerably accelerate learning
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
22
Three Approaches to Q(λ)
❐ How can we extend this to Qlearning?
❐ If you mark every state action
pair as eligible, you backup
over non-greedy policy
Watkins: Zero out
eligibility trace after a nongreedy action. Do max
when backing up at first
non-greedy choice.
%'1 + !"et #1(s, a)
et (s, a) = &
0
'( !"e (s,a)
t #1
if s = st , a = at ,Qt #1(st ,at ) = max a Qt #1(st , a)
if Qt #1(st ,at ) $ max a Qt #1(st ,a)
otherwise
Qt +1(s, a) = Qt (s, a) + )*t et (s, a)
*t = rt +1 + ! max a + Qt (st +1, a +) # Qt (st ,at )
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
23
Watkins’s Q(λ)
Initialize Q(s,a) arbitrarily and e(s, a) = 0, for all s, a
Repeat (for each episode) :
Initialize s, a
Repeat (for each step of episode) :
Take action a, observe r, s!
Choose a ! from s! using policy derived from Q (e.g. ? - greedy)
a* " arg max b Q(s!, b) (if a ties for the max, then a* " a!)
# " r + $Q(s!, a !) % Q(s, a* )
e(s,a) " e(s,a) + 1
For all s,a :
Q(s, a) " Q(s, a) + &#e(s, a)
If a ! = a* , then e(s, a) " $'e(s,a)
else e(s, a) " 0
s " s!;a " a !
Until s is terminal
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
24
Peng’s Q(λ)
❐ Disadvantage to Watkins’s
method:
Early in learning, the
eligibility trace will be
“cut” (zeroed out)
frequently resulting in little
advantage to traces
❐ Peng:
Backup max action except
at end
Never cut traces
❐ Disadvantage:
Complicated to implement
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
25
Naïve Q(λ)
❐ Idea: is it really a problem to
backup exploratory actions?
Never zero traces
Always backup max at
current action (unlike Peng
or Watkins’s)
❐ Is this truly naïve?
❐ Works well is preliminary
empirical studies
What is the backup diagram?
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
26
Comparison Task
❐ Compared Watkins’s, Peng’s, and Naïve (called
McGovern’s here) Q(λ) on several tasks.
See McGovern and Sutton (1997). Towards a Better Q(
λ) for other tasks and results (stochastic tasks,
continuing tasks, etc)
❐ Deterministic gridworld with obstacles
10x10 gridworld
25 randomly generated obstacles
30 runs
α = 0.05, γ = 0.9, λ = 0.9, ε = 0.05, accumulating traces
From McGovern and Sutton (1997). Towards a better Q(λ)
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
27
Comparison Results
From McGovern and Sutton (1997). Towards a better Q(λ)
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
28
Convergence of the Q(λ)’s
❐ None of the methods are proven to converge.
Much extra credit if you can prove any of them.
❐ Watkins’s is thought to converge to Q*
❐ Peng’s is thought to converge to a mixture of Qπ and Q*
❐ Naïve - Q*?
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
29
Eligibility Traces for Actor-Critic Methods
❐ Critic: On-policy learning of Vπ. Use TD(λ) as described
before.
❐ Actor: Needs eligibility traces for each state-action pair.
❐ We change the update equation:
# pt (s,a) + !" t
pt +1(s, a) = $
% pt (s, a)
if a = at and s = st
otherwise
to
pt +1 ( s, a ) = pt ( s, a ) + !" t et ( s, a )
❐ Can change the other actor-critic update:
% pt (s,a) + !" t [1# $ (s, a)] if a = at and s = st
pt +1(s, a) = &
pt (s,a)
otherwise
'
where
to
pt +1 ( s, a ) = pt ( s, a ) + !" t et ( s, a )
%!"et #1(s, a) + 1 # $ t (st ,at ) if s = st and a = at
et (s, a) = &
!"et #1(s, a)
otherwise
'
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
30
Replacing Traces
❐ Using accumulating traces, frequently visited states can
have eligibilities greater than 1
This can be a problem for convergence
❐ Replacing traces: Instead of adding 1 when you visit a
state, set that trace to 1
%!"et #1(s) if s $ st
et (s) = &
if s = st
' 1
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
31
Replacing Traces Example
❐ Same 19 state random walk task as before
❐ Replacing traces perform better than accumulating traces over more
values of λ
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
32
Why Replacing Traces?
❐ Replacing traces can significantly speed learning
❐ They can make the system perform well for a broader set of
parameters
❐ Accumulating traces can do poorly on certain types of tasks
Why is this task particularly onerous
for accumulating traces?
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
33
More Replacing Traces
❐ Off-line replacing trace TD(1) is identical to first-visit MC
❐ Extension to action-values:
When you revisit a state, what should you do with the
traces for the other actions?
Singh and Sutton say to set them to zero:
$&
1
if s = st and a = at
et (s, a) = %
0
if s = st and a ( at
&!"e (s, a)
if s ( st
' t #1
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
34
Implementation Issues
❐ Could require much more computation
But most eligibility traces are VERY close to zero
❐ If you implement it in Matlab, backup is only one line of
code and is very fast (Matlab is optimized for matrices)
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
35
Variable λ
❐ Can generalize to variable λ
if s $ st
% !"t et #1(s)
et (s) = &
'!"t et #1(s) + 1 if s = st
❐ Here λ is a function of time
Could define
"t = " ( st ) or "t = "
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
t
!
36
Conclusions
❐ Provides efficient, incremental way to combine MC and
TD
Includes advantages of MC (can deal with lack of
Markov property)
Includes advantages of TD (using TD error,
bootstrapping)
❐ Can significantly speed learning
❐ Does have a cost in computation
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
37
Something Here is Not Like the Other
R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
38
0
You can add this document to your study collection(s)
Sign in Available only to authorized usersYou can add this document to your saved list
Sign in Available only to authorized users(For complaints, use another form )