Model Validation as
an Integrated Social Process
George P. Richardson
Rockefeller College of Public Affairs and Policy
University at Albany - State University of New York
GPR@Albany.edu
Rockefeller College
of Public Affairs and Policy
1
University at Albany
State University of New York
What do we mean by ‘validation’?
• No model has ever been or ever will be thoroughly
validated. …‘Useful,’ ‘illuminating,’ or ‘inspiring
confidence’ are more apt descriptors applying to models
than ‘valid’ (Greenberger et al. 1976).
• Validation is a process of establishing confidence in the
soundness and usefulness of a model. (Forrester 1973, Forrester
and Senge 1980).
Rockefeller College
of Public Affairs and Policy
2
University at Albany
State University of New York
The classic questions
• Not ‘Is the model valid,’ but
• Is the model suitable for its purposes and the
problem it addresses?
• Is the model consistent with the slice of reality it
tries to capture? (Richardson & Pugh 1981)
Rockefeller College
of Public Affairs and Policy
3
University at Albany
State University of New York
The system dynamics modeling process
Perceptions of
System Structure
Comparison and
Reconcilation
Representation of
Model Structure
System
Conceptualization
Model
Formulation
Empirical and
Inferred Time
Series
Comparison and
Reconciliation.
Deduction Of
Model Behavior
Adapted from Saeed 1992
Rockefeller College
of Public Affairs and Policy
4
University at Albany
State University of New York
Processes focusing on system structure
Mental Models,
Experience,
Literature
Perceptions of
System Structure
Comparison and
Reconcilation
Representation of
Model Structure
Empirical
Evidence
System
Conceptualization
Model
Formulation
Diagramming and
Description Tools
Rockefeller College
of Public Affairs and Policy
5
University at Albany
State University of New York
Processes focusing on system behavior
Empirical
Evidence
System
Conceptualization
Model
Formulation
Rockefeller College
of Public Affairs and Policy
6
Literature,
Experience
Empirical and
Inferred Time
Series
Comparison and
Reconciliation.
Deduction Of
Model Behavior
Computing
Aids
University at Albany
State University of New York
Two kinds of validating processes
Mental Models,
Experience,
Literature
Perceptions of
System Structure
Empirical
Evidence
Literature,
Experience
Empirical and
Inferred Time
Series
System
Conceptualization
Comparison and
Comparison and Structure
Behavior
Reconciliation.
Reconcilation Validating
Validating
Processes
Processes
Model
Formulation
Representation of
Deduction Of
Model Structure
Model Behavior
Computing
Diagramming and
Aids
Description Tools
Rockefeller College
of Public Affairs and Policy
7
University at Albany
State University of New York
The classic tests
Focusing on
STRUCTURE
Focusing on
BEHAVIOR
Testing
SUITABILITY for
PURPOSES
• Dimensional consistency
• Extreme conditions
• Boundary adequacy
• Parameter insensitivity
• Structure insensitivity
Testing
CONSISTENCY with
REALITY
• Face validity
• Parameter values
• Replication of behavior
• Surprise behavior
• Statistical tests
Contributing to
UTILITY &
EFFECTIVENESS
• Appropriateness for
audience
• Counterintuitive behavior
• Generation of insights
Forrester 1973, Forrester & Senge 1980, Richardson and Pugh 1981
Rockefeller College
of Public Affairs and Policy
8
University at Albany
State University of New York
Validation is present at every step
• Conceptualizing:
• Do we have the right people?
• The right dynamic problem definition?
• The right level of aggregation?
•
•
•
•
•
Mapping: Developing promising dynamic hypotheses
Formulating: Clarity, logic, and extremes
Simulating: Right behavior for right reasons
Deciding: Implementable conclusions
Implementing: Requires conviction!
Rockefeller College
of Public Affairs and Policy
9
University at Albany
State University of New York
Do we have the right people?
Rockefeller College
of Public Affairs and Policy
10
University at Albany
State University of New York
Opposition
High
Weak opponents
Strong opponents
Weak supporters
Strong supporters
Low
Low
Support
Problem Frame
Problem frame stakeholder map
High
Weak
Strong
Stakeholder Power
Bryson, Strategic Planning for Public and Nonprofit Organizations
Rockefeller College
of Public Affairs and Policy
11
University at Albany
State University of New York
Power versus Interest grid
High
Players
Crowd
Context setters
Interest
Subjects
Low
Weak
Strong
Power
Eden & Ackerman 1998
Rockefeller College
of Public Affairs and Policy
12
University at Albany
State University of New York
Pursuing validity in mapping
• Think causally, not correlationally
• Think stocks and flows, even if you don’t draw
them
• Use units to make the causal logic plausible, even
if you don’t write them down
• Be able to tell a story for every link and loop
• Move progressively from less precise to more
precise -- from informal map to formal map
Rockefeller College
of Public Affairs and Policy
13
University at Albany
State University of New York
The standard cautions
Understandings
of the system
Understandings
of the model
Carbon in
carnivores
System
conceptualization
Model formulation
& testing
Carbon in
herbivores
Carbon in soil
Carbon in algae,
plants & trees
Prejudice
Achievements of
the minority
Carbon in
atmosphere
Discrimination
Opportunities for
the minority
Rockefeller College
of Public Affairs and Policy
14
University at Albany
State University of New York
These arrows mean ‘and then’
Understandings
of the system
Understandings
of the model
• We start with some understandings of the
Carbon
in
problem and its
systemic
context, and
carnivores
Carbon
in
then we conceptualize (map)
the system.
System
conceptualization
Model formulation
& testing
•
Carbon in
Then
we build
herbivores
the
beginnings of a model,
Carbon in soil
which we then test to understand it.
Carbon in algae,
reformulate,
plants & treesor
• Then we
reconceptualize,
or revise our understandings, or do some
of all three, and then continue…
Prejudice
Achievements of
the minority
atmosphere
Discrimination
Opportunities for
the minority
Rockefeller College
of Public Affairs and Policy
15
University at Albany
State University of New York
Arrows here are flows of material
Understandings
of the system
The words here
Understandings
represent
stocks.
System
of the model
conceptualization
This
is
not
a
Model formulation
& testing
causal
diagram.
Carbon in
carnivores
Carbon in
herbivores
Carbon in soil
Carbon in algae,
plants & trees
Prejudice
Achievements of
the minority
Carbon in
atmosphere
Discrimination
Opportunities for
the minority
Rockefeller College
of Public Affairs and Policy
16
University at Albany
State University of New York
Only this one is a causal loop
Understandings
of the system
No explicit stocks or flows,
no clear units, but it tells a
Understandings
System– It’s a Carbon in
of the
model
compelling
story
conceptualization
herbivores
good start.
Model formulation
& testing
Carbon in
atmosphere
Carbon in soil
Carbon in algae,
plants & trees
Prejudice
Achievements of
the minority
Carbon in
carnivores
Discrimination
Opportunities for
the minority
Rockefeller College
of Public Affairs and Policy
17
University at Albany
State University of New York
Project modeling core structure
Work to
be done
Work in process
beginning
work
completing
work
doing work
incorrectly
starting
rework
Known
rework
Rockefeller College
of Public Affairs and Policy
Work
really done
rework
discovery
18
Undiscovered
rework
University at Albany
State University of New York
Identical structure
without explicit stocks and flows
completing
work
beginning
work
Work in
process
Work to be
done
Work really
done
starting
rework
doing work
incorrectly
Known rework
Undiscovered
rework
rework
discovery
Rockefeller College
of Public Affairs and Policy
19
University at Albany
State University of New York
Pursuing validity writing equations
•
•
•
•
Recognizable parameters
Robust equation forms
Phase relations
Richardson’s Rule: Every complicated, ugly,
excessively mathematical equation and every
equation flaw saps confidence in the model.
Rockefeller College
of Public Affairs and Policy
20
University at Albany
State University of New York
Modeling conflict within & between nations
Adaptation
Potential for
international
conflict
International
conflict
+
Consequences of
conflict
+
-
+
Population
growth rate
Technology
growth rate
of Public Affairs and Policy
+
+
Population
Technology
-
+
Rockefeller College
+
+
Domestic
conflict
Potential for
conflict
Lateral
pressure
Internal
stress +
Domestic
adaptation
21
University at Albany
State University of New York
Complexity & flaws destroy confidence
• P of int'l conflict =
DELAY FIXED ((Lateral pressure/10*Military force
effect/Trade and bargaining leverage + International
conflict)/Lateral conflict break point, 1 , 0)
• Flaws
Complexity, discreteness, units confusion and disagreement,
disembodied parameter, confusion of the effect of a concept
[leverage] with the concept itself, and the wonder what keeps
this probability between 0 and 1?
Rockefeller College
of Public Affairs and Policy
22
University at Albany
State University of New York
Robust equation forms
Progress
Rockefeller College
of Public Affairs and Policy
23
Cumulative
progress
University at Albany
State University of New York
Causal mish-mash
Hours per person
per day
Workers
Workweek
(days)
Normal effectiveness
(tasks/hour)
Progress
Cumulative
progress
Effect of
schedule
pressure
Effect of
motivation
Rockefeller College
of Public Affairs and Policy
Effect of ...
24
University at Albany
State University of New York
Robust equation formulations
Effort
(hours/month)
Progress
Cumulative
progress
Effectiveness
(tasks/hour)
Rockefeller College
of Public Affairs and Policy
25
University at Albany
State University of New York
Robust equation formulations
Hours per person
per day
Workweek
(days)
Workers
Effort
(hours/month)
Progress
Cumulative
progress
Effectiveness
(tasks/hour)
Rockefeller College
of Public Affairs and Policy
26
University at Albany
State University of New York
Robust equation formulations
Effort
(hours/month)
Progress
Normal effectiveness
(tasks/hour)
Cumulative
progress
Effectiveness
(tasks/hour)
Effect of
motivation
Rockefeller College
of Public Affairs and Policy
Effect of
schedule
pressure
Effect of ...
27
University at Albany
State University of New York
Robust equation formulations
Hours per person
per day
Workweek
(days)
Workers
Effort
(hours/month)
Progress
Normal effectiveness
(tasks/hour)
Cumulative
progress
Effectiveness
(tasks/hour)
Effect of
motivation
Rockefeller College
of Public Affairs and Policy
Effect of
schedule
pressure
Effect of ...
28
University at Albany
State University of New York
Pursuing validity in equations: Phasing
(B) Problems
threaten scope
generating
problems
Problems
generated
integrating
info
Problems generated
per info unit integrated
(R ) Problems
compound
Unintegrated
information
Unitegrated
info within
scope
Scope of
integration effort
Ease of
integrating
info
of Public Affairs and Policy
(R ) Success
enhances resources
Effort to
integrating info
(R ) Perceived value
enhances scope
Willingness to cede
control of info
Rockefeller College
(B) Low
hanging fruit
(B) Problems
impede progress
Pressure to allocate
resources elsew here
integrating info
(B) Problems rob
resources
subtracting
resources to
integration
29
Integrated
information
Perceived value
of integrated
information
Resources
devoted to info
integration
adding to resources
to integration
University at Albany
State University of New York
Phase relations
Integrated
information
Constant Perceived Value
suggests continually rising
Resources, but that
doesn’t seem correct
Perceived value
of integrated
information
Resources
devoted to info
integration
adding to resources
to integration
Rockefeller College
of Public Affairs and Policy
30
University at Albany
State University of New York
Phase relations
Perceived value
of integrated
information
Here, the Perceived Value of
Integrated Information sets a
planned level of resources
Resources
allocated to
integration
project
Resources
planned to info
integration
adding to resources
to integration
Time to allocate
resources
Rockefeller College
of Public Affairs and Policy
31
University at Albany
State University of New York
Pursuing validity fitting to data
• Generally, a weak test of model validity
• Whole-model procedures
• Optimization
• Partial-model procedures
• Reporting results
• Graphically
• Numerically: Theil statistics
Rockefeller College
of Public Affairs and Policy
32
University at Albany
State University of New York
Example of weakness of fitting to data
Discovery / production
experience & technology
• Logistic curve
• dx/dt = ax - bx2
(R)
• Gompertz curve
Cumulative
production
Petroleum
production
• dx/dt = ax - bx ln(x)
(B)
Constraints from the
resource remaining
Rockefeller College
of Public Affairs and Policy
33
University at Albany
State University of New York
Fitting global petroleum with Logistic
Production
40,000
40,000
20,000
20,000
Cum Production
2M
2M
0
0
1880 1902 1924 1946 1968 1990 2012 2034 2056 2078 2100
Time (year)
1M
1M
0
0
1880 1902 1924 1946 1968 1990 2012
Time (year)
Rockefeller College
of Public Affairs and Policy
34
2034 2056 2078 2100
University at Albany
State University of New York
Fitting global petroleum with Gompertz
Production
40,000
30,000
20,000
Cum Production
10,000
4M
2M
0
1880 1902 1924 1946 1968 1990 2012 2034 2056 2078 2100
Time (year)
2M
1M
0
0
1880 1902 1924 1946
Rockefeller College
of Public Affairs and Policy
35
1968 1990 2012
Time (year)
2034 2056 2078
2100
University at Albany
State University of New York
Presenting model fit visually
Rockefeller College
of Public Affairs and Policy
36
University at Albany
State University of New York
Presenting model fit numerically
• Theil statistics, for example
• Based on a breakdown of the mean squared error:
2
2
2
(

)
(X

Y
)
s
s
1/ n  i i  (X  Y )  x y  2(1  r) sx sy
• 1 = Bias + Variation + Covariation
Rockefeller College
of Public Affairs and Policy
37
University at Albany
State University of New York
Presenting model fit numerically
Bia s
Va riati on
Covaria tion
RMSPE
RMSEPM
U
r
As sets
0.001
0.001
0.998
0.111
0.119
0.097
0.986
Li abil itie s
0.136
0.426
0.438
0.208
0.099
0.079
0.996
Prem ium inc
0.376
0.047
0.576
10 66.0 00
0.693
0.562
0.664
Surp lus
0.019
0.640
0.341
0.437
-0.178
0.133
0.994
Case s op en
0.119
0.354
0.526
0.049
0.061
0.051
0.998
To tal premi ums
0.094
0.409
0.497
0.173
0.298
0.256
0.908
Rockefeller College
of Public Affairs and Policy
38
University at Albany
State University of New York
Learning from surprise model behavior
• Have clear a priori expectations
• Follow up all unanticipated behavior to
appropriate resolution
• Confirm all behavioral hypotheses through
appropriate model tests (Mass 1991/1981)
Rockefeller College
of Public Affairs and Policy
39
University at Albany
State University of New York
Tests to reveal and resolve surprise behavior
•
•
•
•
•
•
Testing the symmetry of policy response (up and down)
Testing large amplitude versus small amplitude response
Testing policies entering at different points
Testing different patterns of behavior
Isolating uniqueness of equilibrium or steady state
Understanding forces producing equilibrium positions
(Mass 1991/1981)
Rockefeller College
of Public Affairs and Policy
40
University at Albany
State University of New York
Summary
• Modelers, stakeholders, problem experts, and others in
the modeling process pursue validity at every step along
the way.
• We have rigorous traditions guiding model creation,
formulation, exploration, and implications.
• We have a powerful, intimidating battery of tests of
model structure and behavior.
• Model-based conclusions that make it through all this
deserve the confidence of everyone in the process.
Rockefeller College
of Public Affairs and Policy
41
University at Albany
State University of New York
Epilog
• Reason is itself a matter of faith. It is an act of
faith to assert that our thoughts have any relation
to reality. (G.K. Chesterton)
• I have no exquisite reason for’t, but I have
reason good enough. (Sir Andrew, Twelfth Night)
Rockefeller College
of Public Affairs and Policy
42
University at Albany
State University of New York