CSE 4705
HW 4
1. EM Algorithm on a Belief Network
2. Auxiliary function
import math
import matplotlib.pyplot as plt
# original function to be minimized
def f(x):
return math.log(math.cosh(x))
# derives the update rule from (e)
def update_rule(x_n):
return x_n - math.tanh(x_n)
history_neg = [-2]
history_pos = [3]
# xsub0 = -2
x_n = -2
for n in range(10):
x_n = update_rule(x_n)
history_neg.append(x_n)
# xsub0 = 3
x_n = 3
for n in range(10):
x_n = update_rule(x_n)
history_pos.append(x_n)
plt.figure(figsize=(10, 6))
plt.plot(history_neg, 'bo-', label='beginning from $x_0 = -2$')
plt.plot(history_pos, 'ro-', label='beginning from $x_0 = 3$')
plt.xlabel('iteration (n)')
plt.ylabel('value ($x_n$)')
plt.legend()
plt.grid(True)
plt.show()
3. Mixture model decision boundary
4. Markov Decision Processes (MDPs)
(a)
•
•
•
•
•
(b)
S (States): the set of finite states the agent can be in, such as their coordinates.
A (Actions): the set of finite actions the agent can act upon, such as moving up or
down in the grid world.
P (Transition probability, P(s’|s, a): the probability of the agent moving to state s’
from state s and taking action a.
R (Reward function, R(s, a, s’): this gives the agent an immediate scalar reward for a
transition.
γ (Discount factor, 0 ≤ γ ≤ 1): prioritizes an immediate reward over future rewards.
A policy maps states to actions. An example of a policy would be to move right at
coordinates (2, 2). A value function is the expected return when starting in state and
following policy π. Let’s the value of the goal state is +5. If the agent is just left of the goal
by one space, and the policy is to move right at this space, the reward for the move would
be +4 = -1 + 5. In simpler terms, a policy is the behavior of the agent, while a value function
gives a measurement of value for that behavior.
(c)
The Bellman equation for the value of a state under a given policy π is the sum of the
probability of taking action a in state s times the probability of transitioning to s’ and
receiving reward r times the expected return from this transition. Its recursive relationship
allows value estimates to be computed iteratively. It determines the value of the current
state based on the values of possible future states.
5. Imitation Learning: Behavior Cloning
(a)
Behavior cloning collects data from an “expert” who performs a task to collect data. Then,
the data is used to train a model that maps states to actions. A model that can be trained
on is a Convolutional Neural Network for autonomous driving, and its objective is to
minimize error between the projected steering angle and the expert driver’s actual angle.
(b)
The main limitation of behavior cloning is that it is only trained on what it learned from the
expert. If the agent makes a mistake, it can be in a state that the expert never visited. The
agent will not know how to resolve this issue. An alternative approach is reinforcement
learning, as instead of learning from an expert, the agent learns based on its own
experiences from the environment.