Using the Method of Simulated Moments (MSM)
for Identification of Dynamic Models
Presenter:
Advisor:
Mohammad S. Jalali
Dr. Hazhir Rahmandad
25th MIT-UAlbany PhD Colloquium, System Dynamics Society
Nov 16, 2012
Introduction
Contents:
Increasingly, dynamic modelers face problems where
estimating model parameters from numerical empirical
data is a requirement.
The field of system dynamics (SD) was ahead of many
approaches available in social sciences in using advanced
analytical methods of the time (Forrester, 1961), and kept
this edge for many years.
For example advanced filtering and estimation methods
were introduced into the SD literature in the 70s (Peterson,
1975).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
2
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Introduction
Contents:
Increasingly, dynamic modelers face problems where
estimating model parameters from numerical empirical
data is a requirement.
The field of system dynamics (SD) was ahead of many
approaches available in social sciences in using advanced
analytical methods of the time (Forrester, 1961), and kept
this edge for many years.
For example advanced filtering and estimation methods
were introduced into the SD literature in the 70s (Peterson,
1975).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
3
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Introduction
Contents:
Increasingly, dynamic modelers face problems where
estimating model parameters from numerical empirical
data is a requirement.
The field of system dynamics (SD) was ahead of many
approaches available in social sciences in using advanced
analytical methods of the time (Forrester, 1961), and kept
this edge for many years.
For example advanced filtering and estimation methods
were introduced into the SD literature in the 70s (Peterson,
1975).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
4
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Introduction
Contents:
However, over the last three decades the research in SD
has largely focused on diverse applications of the original
toolbox, with limited methodological expansions in
parameter estimation domain.
In contrast, research in econometrics and other related
fields have provided many relevant tools over this period
(Greene, 2012).
As a result, currently many studies in SD do not report
formal parameter estimates common in social science
research, or when calibration is pursued, typical measures
of confidence in estimated parameter values are not
reported.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
5
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Introduction
Contents:
However, over the last three decades the research in SD
has largely focused on diverse applications of the original
toolbox, with limited methodological expansions in
parameter estimation domain.
In contrast, research in econometrics and other related
fields have provided many relevant tools over this period
(Greene, 2012).
As a result, currently many studies in SD do not report
formal parameter estimates common in social science
research, or when calibration is pursued, typical measures
of confidence in estimated parameter values are not
reported.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
6
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Introduction
Contents:
However, over the last three decades the research in SD
has largely focused on diverse applications of the original
toolbox, with limited methodological expansions in
parameter estimation domain.
In contrast, research in econometrics and other related
fields have provided many relevant tools over this period
(Greene, 2012).
As a result, currently many studies in SD do not report
formal parameter estimates common in social science
research, or when calibration is pursued, typical measures
of confidence in estimated parameter values are not
reported.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
7
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Common Techniques
Contents:
Modelers typically compare time series data against the
same variables in a model,
and
minimize the weighted sum of a function of the error term
by
changing the uncertain parameters until best fitting
estimates are found through a nonlinear optimization
algorithm (Oliva, 2003).
The error function is frequently defined as the squared
error but absolute error and absolute percent error terms
are also common (Sterman, 2000).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
8
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Common Techniques
Contents:
Modelers typically compare time series data against the
same variables in a model,
and
minimize the weighted sum of a function of the error term
by
changing the uncertain parameters until best fitting
estimates are found through a nonlinear optimization
algorithm (Oliva, 2003).
The error function is frequently defined as the squared
error but absolute error and absolute percent error terms
are also common (Sterman, 2000).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
9
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Common Techniques
Contents:
Weights for different data points are given based on the
confidence the researcher has in the accuracy of the data
and its relevance to the problem at hand.
When reported, confidence intervals are calculated using
normality and independence assumption for error terms
which,
with weights proportional to the reciprocal of error
variance, would turn least squared error estimates into
maximum-likelihood estimates (MLE).
Bootstrapping methods are also sometimes used for
estimating confidence intervals (Dogan, 2007).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
10
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Common Techniques
Contents:
Weights for different data points are given based on the
confidence the researcher has in the accuracy of the data
and its relevance to the problem at hand.
When reported, confidence intervals are calculated using
normality and independence assumption for error terms
which,
with weights proportional to the reciprocal of error
variance, would turn least squared error estimates into
maximum-likelihood estimates (MLE).
Bootstrapping methods are also sometimes used for
estimating confidence intervals (Dogan, 2007).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
11
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Common Techniques
Contents:
Weights for different data points are given based on the
confidence the researcher has in the accuracy of the data
and its relevance to the problem at hand.
When reported, confidence intervals are calculated using
normality and independence assumption for error terms
which,
with weights proportional to the reciprocal of error
variance, would turn least squared error estimates into
maximum-likelihood estimates (MLE).
Bootstrapping methods are also sometimes used for
estimating confidence intervals (Dogan, 2007).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
12
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Some Challenges
Contents:
Challenges with the current parameter estimation
techniques?
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
13
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Some Challenges
Contents:
While the common approaches cover many important
estimation challenges, they each include some
shortcomings.
Ad hoc selection of the error term and the weights for
different data points reduces the consistency of the
methods and their ability to provide confidence intervals.
Normality and independence may regularly be violated
which negate the benefits of MLE when using squared
errors.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
14
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Some Challenges
Contents:
While the common approaches cover many important
estimation challenges, they each include some
shortcomings.
Ad hoc selection of the error term and the weights for
different data points reduces the consistency of the
methods and their ability to provide confidence intervals.
Normality and independence may regularly be violated
which negate the benefits of MLE when using squared
errors.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
15
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Some Challenges
Contents:
While the common approaches cover many important
estimation challenges, they each include some
shortcomings.
Ad hoc selection of the error term and the weights for
different data points reduces the consistency of the
methods and their ability to provide confidence intervals.
Normality and independence may regularly be violated
which negate the benefits of MLE when using squared
errors.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
16
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Some Challenges
Contents:
Bootstrapping, while flexible, increases the computational
costs significantly and as a result may prove infeasible for
many realistically-sized problems.
All these methods rely on having time series data, and
cannot extract from distributions in cross-sectional data
the dynamics that have led to those distributions.
Note: Cross-sectional data is data collected at the same point of time.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
17
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Some Challenges
Contents:
Bootstrapping, while flexible, increases the computational
costs significantly and as a result may prove infeasible for
many realistically-sized problems.
All these methods rely on having time series data, and
cannot extract from distributions in cross-sectional data
the dynamics that have led to those distributions.
Note: Cross-sectional data is data collected at the same point of time.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
18
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Estimation Procedures’ Requirements
Contents:
The following characteristics signify the estimation
procedures ideal for dynamic modelers:
Model Independence: given that most dynamic models
do not follow a fixed structural form (e.g. linearity),
estimation procedures that are independent of model
structure are most beneficial.
Analytical confidence intervals: Ability to find
confidence intervals analytically is important because of
the computational costs of optimizing none-linear dynamic
models and replications needed for bootstrapping
methods.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
19
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Estimation Procedures’ Requirements
Contents:
The following characteristics signify the estimation
procedures ideal for dynamic modelers:
Model Independence: given that most dynamic models
do not follow a fixed structural form (e.g. linearity),
estimation procedures that are independent of model
structure are most beneficial.
Analytical confidence intervals: Ability to find
confidence intervals analytically is important because of
the computational costs of optimizing none-linear dynamic
models and replications needed for bootstrapping
methods.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
20
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Estimation Procedures’ Requirements
Contents:
The following characteristics signify the estimation
procedures ideal for dynamic modelers:
Model Independence: given that most dynamic models
do not follow a fixed structural form (e.g. linearity),
estimation procedures that are independent of model
structure are most beneficial.
Analytical confidence intervals: Ability to find
confidence intervals analytically is important because of
the computational costs of optimizing none-linear dynamic
models and replications needed for bootstrapping
methods.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
21
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Estimation Procedures’ Requirements
Contents:
The Assumption-free error terms: Independence and
distributional assumptions on error terms for dynamic
models are not always easy to justify, so methods with
fewer such assumptions are preferred.
Applicability to diverse data types: including both time
series and cross-sectional data.
No single method fully satisfies all these requirements,
therefore modelers need to choose from a menu of
available estimation methods to match their problem
requirements.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
22
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Estimation Procedures’ Requirements
Contents:
The Assumption-free error terms: Independence and
distributional assumptions on error terms for dynamic
models are not always easy to justify, so methods with
fewer such assumptions are preferred.
Applicability to diverse data types: including both time
series and cross-sectional data.
No single method fully satisfies all these requirements,
therefore modelers need to choose from a menu of
available estimation methods to match their problem
requirements.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
23
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Estimation Procedures’ Requirements
Contents:
The Assumption-free error terms: Independence and
distributional assumptions on error terms for dynamic
models are not always easy to justify, so methods with
fewer such assumptions are preferred.
Applicability to diverse data types: including both time
series and cross-sectional data.
No single method fully satisfies all these requirements,
therefore modelers need to choose from a menu of
available estimation methods to match their problem
requirements.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
24
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Method of Simulated Moments
The basic idea of this method is to define appropriate
moments of data and, by changing uncertain parameters,
minimize the difference between those moments and their
simulated counterpart coming from the model.
Moments could be any function of data points available.
However for analytical confidence intervals to be available,
one needs these moments to be normally distributed,
often meaning that:
each moment is an average across a function of multiple
independent observations coming from the same
underlying distribution (then normality follows from the central limit theorem).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
25
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Method of Simulated Moments
The basic idea of this method is to define appropriate
moments of data and, by changing uncertain parameters,
minimize the difference between those moments and their
simulated counterpart coming from the model.
Moments could be any function of data points available.
However for analytical confidence intervals to be available,
one needs these moments to be normally distributed,
often meaning that:
each moment is an average across a function of multiple
independent observations coming from the same
underlying distribution (then normality follows from the central limit theorem).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
26
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Method of Simulated Moments
The basic idea of this method is to define appropriate
moments of data and, by changing uncertain parameters,
minimize the difference between those moments and their
simulated counterpart coming from the model.
Moments could be any function of data points available.
However for analytical confidence intervals to be available,
one needs these moments to be normally distributed,
often meaning that:
each moment is an average across a function of multiple
independent observations coming from the same
underlying distribution (then normality follows from the central limit theorem).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
27
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
History of MSM
McFadden (1989) was the first who proposed using
simulation instead of trying to solve the moment conditions
analytically.
He believes that an unbiased simulator is used in the MSM
where the simulation errors are independent across
observations, and the variance introduced by simulation
will be controlled by the law of large numbers operating
across observations.
Lee (1991) and Duffie (1993) extended the framework and
provided a rigorous treatment of MSM estimators for timeseries and panel-data cases and provided relevant
statistics for making tests.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
28
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
History of MSM
McFadden (1989) was the first who proposed using
simulation instead of trying to solve the moment conditions
analytically.
He believes that an unbiased simulator is used in the MSM
where the simulation errors are independent across
observations, and the variance introduced by simulation
will be controlled by the law of large numbers operating
across observations.
Lee (1991) and Duffie (1993) extended the framework and
provided a rigorous treatment of MSM estimators for timeseries and panel-data cases and provided relevant
statistics for making tests.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
29
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
History of MSM
McFadden (1989) was the first who proposed using
simulation instead of trying to solve the moment conditions
analytically.
He believes that an unbiased simulator is used in the MSM
where the simulation errors are independent across
observations, and the variance introduced by simulation
will be controlled by the law of large numbers operating
across observations.
Lee (1991) and Duffie (1993) extended the framework and
provided a rigorous treatment of MSM estimators for timeseries and panel-data cases and provided relevant
statistics for making tests.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
30
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
When to Use MSM
Contents:
MSM is best fitted for dynamic modeling problems when
some of the following problem characteristics are present:
Population data:
MSM is suitable for estimation of generic models to
population data. Different units of data such as individuals,
firms, and countries could be available. For each unit one
or more data items (e.g. weight, height, and age for
individual data) could be available.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
31
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
When to Use MSM
Contents:
MSM is best fitted for dynamic modeling problems when
some of the following problem characteristics are present:
Population data:
MSM is suitable for estimation of generic models to
population data. Different units of data such as individuals,
firms, and countries could be available. For each unit one
or more data items (e.g. weight, height, and age for
individual data) could be available.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
32
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
When to Use MSM
Contents:
Role of random processes:
MSM could be a good choice when models include
stochastic processes that drive the model, and their impact
on the model behavior is reflected in the data against
which the model is to be calibrated, e.g. when we are
trying to match the variance observed across multiple
units.
Confidence Intervals:
MSM would be a suitable choice when analytical
confidence intervals are sought.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
33
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
When to Use MSM
Contents:
Role of random processes:
MSM could be a good choice when models include
stochastic processes that drive the model, and their impact
on the model behavior is reflected in the data against
which the model is to be calibrated, e.g. when we are
trying to match the variance observed across multiple
units.
Confidence Intervals:
MSM would be a suitable choice when analytical
confidence intervals are sought.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
34
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
When to Use MSM
Contents:
Cross-sectional data:
MSM applies to both cross-sectional and time series data.
Where-as time series data include multiple data points for
the same unit over time, cross-sectional data includes data
points for multiple units at the same time.
MSM may be the only viable choice for estimating dynamic
models when data is cross-sectional as it allows us to
extract the information about the historical trajectories of
units hidden in their cross-sectional distributions.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
35
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
When to Use MSM
Contents:
Cross-sectional data:
MSM applies to both cross-sectional and time series data.
Where-as time series data include multiple data points for
the same unit over time, cross-sectional data includes data
points for multiple units at the same time.
MSM may be the only viable choice for estimating dynamic
models when data is cross-sectional as it allows us to
extract the information about the historical trajectories of
units hidden in their cross-sectional distributions.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
36
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
Usually the first and second moments of model’s
outcomes (mean and variance) are good candidates.
Remember that the number of moment conditions should
be (equal to or) larger than the number of unknown
parameters.
The most informative moments are the ones that
1) Are sensitive to at least one of the unknown
parameters
2) Have rather small variances.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
37
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
Usually the first and second moments of model’s
outcomes (mean and variance) are good candidates.
Remember that the number of moment conditions should
be (equal to or) larger than the number of unknown
parameters.
The most informative moments are the ones that
1) Are sensitive to at least one of the unknown
parameters
2) Have rather small variances.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
38
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
Usually the first and second moments of model’s
outcomes (mean and variance) are good candidates.
Remember that the number of moment conditions should
be (equal to or) larger than the number of unknown
parameters.
The most informative moments are the ones that
1) Are sensitive to at least one of the unknown
parameters
2) Have rather small variances.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
39
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
The right choice of moment conditions is the most crucial
step in identifying the model and recovering model
parameters.
Identification of a model using MSM requires that the
model-generated moment conditions should fit their
empirical counterparts if and only if the structural
parameters equal their true values.
Furthermore, the sufficient condition for identification is a
one-to-one mapping between the structural parameters
and a subset of the moment restrictions of the same
dimension.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
40
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
The right choice of moment conditions is the most crucial
step in identifying the model and recovering model
parameters.
Identification of a model using MSM requires that the
model-generated moment conditions should fit their
empirical counterparts if and only if the structural
parameters equal their true values.
Furthermore, the sufficient condition for identification is a
one-to-one mapping between the structural parameters
and a subset of the moment restrictions of the same
dimension.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
41
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
The right choice of moment conditions is the most crucial
step in identifying the model and recovering model
parameters.
Identification of a model using MSM requires that the
model-generated moment conditions should fit their
empirical counterparts if and only if the structural
parameters equal their true values.
Furthermore, the sufficient condition for identification is a
one-to-one mapping between the structural parameters
and a subset of the moment restrictions of the same
dimension.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
42
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
- The moment represented by the dashed–line is not informative.
- It is not very sensitive to changes in parameter value.
- We can not even be sure that these small changes are due to true
response of the model to various parameter value or are the artifact of
computational or sampling errors.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
43
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
Example:
A simple model of individual’s body mass, consisting of fat
mass and fat free mass, is developed by Rahmandad
(2011).
The model included a few uncertain parameters. In
absence of time series data, those parameters were
estimated from cross sectional data on individual weights
coming from the National Health and Nutrition Examination
Survey (NHANES).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
44
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
Example:
A simple model of individual’s body mass, consisting of fat
mass and fat free mass, is developed by Rahmandad
(2011).
The model included a few uncertain parameters. In
absence of time series data, those parameters were
estimated from cross sectional data on individual weights
coming from the National Health and Nutrition Examination
Survey (NHANES).
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
45
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
Example:
NHANES 2005-2006 population of 5,971 subjects was
categorized into 110 subpopulations based on:
different ethnicities (5 ethnicities),
genders (2 genders),
age (11 age groups).
For each population group two moments, average body
weight and variance of body weight, were calculated as
the moments to be matched, leading to a total of 220
moments to match.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
46
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 1: Choose the Moment Conditions
Example:
NHANES 2005-2006 population of 5,971 subjects was
categorized into 110 subpopulations based on:
different ethnicities (5 ethnicities),
genders (2 genders),
age (11 age groups).
For each population group two moments, average body
weight and variance of body weight, were calculated as
the moments to be matched, leading to a total of 220
moments to match.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
47
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 2: Weighting Matrix
A two-step procedure should be developed to calculate the
optimal weighting matrix (W) (More technical information will be presented
in the forthcoming book chapter.)
The right choice of weighting matrix ensures that we get
the smallest asymptotic variance for our estimated values.
In the simplest case, the identity matrix is used as the
initial value of the weighting matrix in the first step.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
48
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 2: Weighting Matrix
A two-step procedure should be developed to calculate the
optimal weighting matrix (W) (More technical information will be presented
in the forthcoming book chapter.)
The right choice of weighting matrix ensures that we get
the smallest asymptotic variance for our estimated values.
In the simplest case, the identity matrix is used as the
initial value of the weighting matrix in the first step.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
49
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 2: Weighting Matrix
A two-step procedure should be developed to calculate the
optimal weighting matrix (W) (More technical information will be presented
in the forthcoming book chapter.)
The right choice of weighting matrix ensures that we get
the smallest asymptotic variance for our estimated values.
In the simplest case, the identity matrix is used as the
initial value of the weighting matrix in the first step.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
50
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 2: Weighting Matrix
A two-step procedure should be developed to calculate the
optimal weighting matrix (W) (More technical information will be presented
in the forthcoming book chapter.)
The right choice of weighting matrix ensures that we get
the smallest asymptotic variance for our estimated values.
In the simplest case, the identity matrix is used as the
initial value of the weighting matrix in the first step.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
51
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 3: Initial Value and Simulation
We need to generate a large sample of model outcome as
the base for estimating model’s moments.
In equilibrium models, it is recommended that we discard
earlier observations (in the time-series sense) since these
observations are sensitive to the initial value of the model
and are not in the steady-state.
If we need observations over time, it is recommended to
generate a vector of simulations and discard the first
observation.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 52
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 3: Initial Value and Simulation
We need to generate a large sample of model outcome as
the base for estimating model’s moments.
In equilibrium models, it is recommended that we discard
earlier observations (in the time-series sense) since these
observations are sensitive to the initial value of the model
and are not in the steady-state.
If we need observations over time, it is recommended to
generate a vector of simulations and discard the first
observation.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 53
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 3: Initial Value and Simulation
We need to generate a large sample of model outcome as
the base for estimating model’s moments.
In equilibrium models, it is recommended that we discard
earlier observations (in the time-series sense) since these
observations are sensitive to the initial value of the model
and are not in the steady-state.
If we need observations over time, it is recommended to
generate a vector of simulations and discard the first
observation.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 54
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 3: Initial Value and Simulation
We need to generate a large sample of model outcome as
the base for estimating model’s moments.
In equilibrium models, it is recommended that we discard
earlier observations (in the time-series sense) since these
observations are sensitive to the initial value of the model
and are not in the steady-state.
If we need observations over time, it is recommended to
generate a vector of simulations and discard the first
observation.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 55
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 4: Optimization Routine and Iteration
This is the most computationally-challenging step of MSM
procedure. We need to minimize the weighted distance of
model-generated moments from empirical moments.
𝛽 = 𝑎𝑟𝑔𝑚𝑖𝑛𝛽∈𝑅 𝑘
1
ℎ 𝑑 −
𝑁
′
𝑁
ℎ 𝑦𝑗 𝛽
𝑗 =1
1
. 𝑊 −1 . ℎ 𝑑 −
𝑁
𝑁
ℎ 𝑦𝑗 𝛽
𝑗 =1
N is the number of simulations,
d is the vector of data,
h is the moment function of interest,
yj(B) is the vector of simulated values corresponding to vector d.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 56
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 4: Optimization Routine and Iteration
This is the most computationally-challenging step of MSM
procedure. We need to minimize the weighted distance of
model-generated moments from empirical moments.
𝛽 = 𝑎𝑟𝑔𝑚𝑖𝑛𝛽∈𝑅 𝑘
1
ℎ 𝑑 −
𝑁
′
𝑁
ℎ 𝑦𝑗 𝛽
𝑗 =1
1
. 𝑊 −1 . ℎ 𝑑 −
𝑁
𝑁
ℎ 𝑦𝑗 𝛽
𝑗 =1
N is the number of simulations,
d is the vector of data,
h is the moment function of interest,
yj(B) is the vector of simulated values corresponding to vector d.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 57
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 4: Optimization Routine and Iteration
We need to use numerical optimization routines to find the
minimum of the total error function. A smart choice of initial
values for parameters may facilitate the quicker
convergence of the optimization routine significantly.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 58
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 5: Interpretation and Making Conclusions
The results we will get from previous steps are just a point
estimation of true parameter values. We need to do further
statistical and computational analysis to make sure that
our results are robust and reliable.
The first step is to look at the confidence interval of
estimated values. The confidence interval estimation will
tell us about the range of probable parameter values.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 59
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 5: Interpretation and Making Conclusions
The results we will get from previous steps are just a point
estimation of true parameter values. We need to do further
statistical and computational analysis to make sure that
our results are robust and reliable.
The first step is to look at the confidence interval of
estimated values. The confidence interval estimation will
tell us about the range of probable parameter values.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 60
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Recipe for MSM
Contents:
Step 5: Interpretation and Making Conclusions
The results we will get from previous steps are just a point
estimation of true parameter values. We need to do further
statistical and computational analysis to make sure that
our results are robust and reliable.
The first step is to look at the confidence interval of
estimated values. The confidence interval estimation will
tell us about the range of probable parameter values.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 61
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
Adda (2003) believed that for implementing the MSM, a
large number of simulations is needed to compute the
standard errors of the estimator.
Using Monte-Carlo analysis, Ruge-Murcia (2012)
presented that even when simulated series are short, the
MSM works and it can deliver accurate estimates;
however, the distribution of the estimates is not efficiently
approximated by the asymptotic normal distribution.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 62
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
Adda (2003) believed that for implementing the MSM, a
large number of simulations is needed to compute the
standard errors of the estimator.
Using Monte-Carlo analysis, Ruge-Murcia (2012)
presented that even when simulated series are short, the
MSM works and it can deliver accurate estimates;
however, the distribution of the estimates is not efficiently
approximated by the asymptotic normal distribution.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 63
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
There are ways to address some of these concerns.
- Providing graphs showing the sensitivity of the
likelihood function to parameter values is one way to
convince the others that the moments were indeed
informative.
- Providing the data (if they are not propriety) and the
programming codes enables others to check the
results independently.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 64
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
There are ways to address some of these concerns.
- Providing graphs showing the sensitivity of the
likelihood function to parameter values is one way to
convince the others that the moments were indeed
informative.
- Providing the data (if they are not propriety) and the
programming codes enables others to check the
results independently.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 65
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
There are ways to address some of these concerns.
- Providing graphs showing the sensitivity of the
likelihood function to parameter values is one way to
convince the others that the moments were indeed
informative.
- Providing the data (if they are not propriety) and the
programming codes enables others to check the
results independently.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 66
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
MLE requires distributional assumptions on error terms.
When such assumptions are justified (either empirically,
based on the observed error, or based on theoretical
considerations), MLE provides a more efficient estimation
method than MSM.
When error terms do not follow any well-established
distribution, MSM may be a better choice.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
67
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
MLE requires distributional assumptions on error terms.
When such assumptions are justified (either empirically,
based on the observed error, or based on theoretical
considerations), MLE provides a more efficient estimation
method than MSM.
When error terms do not follow any well-established
distribution, MSM may be a better choice.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
68
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
MSM provides a large degrees freedom for choosing the
moment conditions which serve the interests of the
modeler.
By imposing different moment conditions one may recover
different structural parameters and as long as there are
enough meaningful moment conditions the researcher has
the luxury of using an arbitrary subset of moments.
This critic compares MSM method to standard
econometrics techniques which is more easily available for
outsiders and provide a better chance of independent
verification of the estimation results.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 69
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
MSM provides a large degrees freedom for choosing the
moment conditions which serve the interests of the
modeler.
By imposing different moment conditions one may recover
different structural parameters and as long as there are
enough meaningful moment conditions the researcher has
the luxury of using an arbitrary subset of moments.
This critic compares MSM method to standard
econometrics techniques which is more easily available for
outsiders and provide a better chance of independent
verification of the estimation results.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 70
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Discussions
Contents:
MSM provides a large degrees freedom for choosing the
moment conditions which serve the interests of the
modeler.
By imposing different moment conditions one may recover
different structural parameters and as long as there are
enough meaningful moment conditions the researcher has
the luxury of using an arbitrary subset of moments.
This critic compares MSM method to standard
econometrics techniques which is more easily available for
outsiders and provide a better chance of independent
verification of the estimation results.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 71
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
References
•
•
•
•
•
•
•
•
•
•
•
•
Adda, J., & Cooper, Russell W. (2003). Dynamic economics : quantitative methods and applications.
Cambridge, Mass.: MIT Press.
Dogan, G. (2007). Bootstrapping for confidence interval estimation and hypothesis testing for parameters
of system dynamics models. System Dynamics Review, 23(4), 415-436.
Duffie, D., & Singleton, K. J. (1993). Simulated Moments Estimation of Markov-Models of Asset Prices.
Econometrica, 61(4), 929-952.
Forrester, Jay W. (1961). Industrial Dynamics (1 ed.). Cambridge: The M.I.T. Press.
Greene, William H. (2012). Econometric analysis (7th ed.). Boston: Prentice Hall.
Lee, Bong-Soo, & Ingram, Beth Fisher. (1991). Simulation estimation of time-series models. Journal of
Econometrics, 47(2–3), 197-205. doi: 10.1016/0304-4076(91)90098-X
Mcfadden, D. (1989). A Method of Simulated Moments for Estimation of Discrete Response Models
without Numerical-Integration. Econometrica, 57(5), 995-1026.
Oliva, R. (2003). Model calibration as a testing strategy for system dynamics models. European Journal
of Operational Research, 151(3), 552-568. doi: Doi 10.1016/S0377-2217(02)00622-7
Peterson, D.W. (1975). Hypothesis, estimation, and validation of dynamic social models- Energy demand
modeling. (Ph.D.), Massachusetts Institute of Technology, Cambridge, MA.
Rahmandad, H., & Sabounchi, N. (2011). Building and estimating a dynamic model of weight gain and
loss for individuals and populations. Paper presented at the the 29th International Conference of the
System Dynamics Society, Washington, DC.
Ruge-Murcia, Francisco. (2012). Estimating nonlinear DSGE models by the simulated method of
moments: With an application to business cycles. Journal of Economic Dynamics and Control, 36(6), 914938. doi: 10.1016/j.jedc.2012.01.008
Sterman, John. (2000). Business dynamics: systems thinking and modeling for a complex world (1 ed.).
Boston: McGraw-Hill/Irwin.
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 72
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Contents:
- Introduction
- Common
Techniques
- Some
Challenges
- Estimation
Procedures’
Requirements
- MSM
- History of
MSM
- When to Use
MSM
- Recipe for
MSM
- Discussions
Thank you!
Any suggestion?
mj @ vt . edu
25th MIT-UAlbany System Dynamics Ph.D. Colloquium, Nov 2012
Slide: 73