Adaptive Stochastic Control for the Smart grid
Qinghua Shen
Smart grid meeting
Outline
• Introduction to the control of smart grid
adaptive stochastic control, smart grid
• Adaptive Stochastic Control
basis of stochastic system, policy search and approximation, convergence
• Example: distributed generation despatch with storage
ADP for resource allocation, value function approximations
• Challenges
1
Introduction to the control of smart grid
• Control of the smart grid
 goal for control: instantly, corrective and dynamically
 Self-healing: auto repair or removal of potentially faulty
equipment
 Flexible: rapid and safe interconnection of distributed
generation and storage
 Predictive: statistics, machine learning, predictive models
 interactive: appropriate information is provided transparently
in near real time
 Optimal : operators and customers efficiently and economically
 Secure : cyber- and physical-security
2
Introduction to the control of smart grid
• Major Components
3
Outline
• Introduction to the control of smart grid
adaptive stochastic control, smart grid
• Adaptive Stochastic Control
basis of stochastic system, policy search and approximation, convergence
• Example: distributed generation despatch with storage
ADP for resource allocation, value function approximations
• Challenges
4
Adaptive Stochastic Control
• Stochastic system
 State variables
 physical state: energy amount, status of a generator
 information state: current and historical demand, price and weather
 belief state: probability distributions
 The decisions
whether or charge/discharge, use backup
 The exogenous Information
all the dimensions of uncertainty
 The Transition Function
given the state, decisions and exogenous information, determines next state
 The Objective Function
metrics that governs how we make those decisions and evaluate the
performance of policies of the controller designs
5
Adaptive Stochastic Control
• Policies

maps the information in state S to a decision x.
, which is the state variables, capturing energy resources Rt,
exogenous information pt, and belief state Kt.
 The problem
is known variously as the value of the a policy or the cost to go function.
can be a cost function if we minimize, or contribution function if we
maximize.
Cost include generating electricity, purchasing fuel, losses due to energy
conversion, cost of repair, and penalties for curtailing loads
 policy for what
includes whether to charge/discharge, when to run a distributed generator,
how much energy draw from grid for every customer in every networks and
the utility
6
Adaptive Stochastic Control
• Design a robust policy: four classes
 Myopic Policies
minimize next-period cost without decisions for future( special structure good)
 Look-ahead Policies
Optimize over some time horizon using a forecast of the possible variability of
exogenous events such as weather. Forecast can either be deterministic
forecasts or stochastic forecasts
 Policy Function Approximations
Functions return an action given a state, without solving any form of
optimization, including: rule-based lookup table; Parameterized
rules(threshold hold); statistical functions
 Policy based on Value Function Approximations
Optimal policy obtained from HJB equation, to avoid curses of dimensionality
a) Approximate to eliminate the expectation; b) replace the value function with
a computationally tractable approximation; c) solve the resulting deterministic
maximization problem using a commercial solver
7
Adaptive Stochastic Control
• ADP and the Post–Decision State
 Value function approximation
when structure of a policy is not obvious, estimates the value of being in a state
When x is a vector, solve the maximization problem is problematic(expectation
hard to compute exactly)---refer to stochastic search
 Post-Decision state
Post decision state determined through current state and action
8
Adaptive Stochastic Control
• Design policy
 look up tables
 Parametric models
With this strategy, we face the challenge of first identifying the basis functions,
and then tuning the parameters
 Nonparametric models
handle high-dimensional, asymptotically unbiased
• Kernel regression;
• Support Vector regression;
• Neural networks;
• Dirichlet process mixtures.
9
Adaptive Stochastic Control
• Policy search
 Direct policy search
Depend on Monte Carlo sampling ----stochastic search
Methods: sequential kriging
Using the knowledge gradient
Applied when the policy structure is apparent
 Bellman residual minimization for value function approximations
This is the most widely used strategy for optimizing policies, and encompasses
a variety of algorithmic approaches that include approximate value iteration
(including temporal difference learning) and approximate policy iteration
10
ASC for distributed generation despatch
• Approximate Dynamic Programming for resource allocation
 Resource allocation
how much energy to store in a battery, whether a diesel generator should be
turned on, and whether a mobile storage device (and/or generator) should be
moved to a congested location.
 A general model
Rta is the number of resources with attribute vector a
xtad is the number of resrouces we act on with a decision
of type d.
a decision d can be (-1,0,1) to discharge, hold, or recharge a battergy
(0,1) to turn a distributed generator off or on
11
Adaptive Stochastic Control
• Value Function Approximations for resource allocation
 Approximate the value
function
resource allocation utility
function: concavity property
Approximate value function
by the post decision resource
vector Separable piece-wise
linear function
Estimate piecewise linear
concave functions by
iteratively stepping forward
through time and updating
value functions
12
Adaptive Stochastic Control
• Experimental work
 evaluate the results
Resource determine the
quality of the resulting policy
is a major challenge
fit the value functions for a
deterministic problem, and
compare the resulting
solution to the optimal
solution for the deterministic
problem, obtained by using a
commercial solver
 limited by the size
13
Challenges
 Convergence
Only some structure can be proofed to be convergence with approximation
Concavity is an important category
 For smart grid
Beneficial to both utility and end users– enough incentive
The track of key performance metrics
14