Asking better
questions to build
better models
Sharon Dawes
Some policy questions
•
•
•
•
•
•
•
Should we eradicate polio?
How can schools help third graders become good readers?
What kinds of public investments lead to urban vitality?
What is the optimal approach to immigration?
How does health insurance coverage affect health care costs?
Can a ban on large sugary drinks reduce obesity and diabetes?
Does public posting of sanitary inspections lead to cleaner
restaurants?
• ...
All models are wrong,
some are useful.
Box, 1979
Why models are wrong
Complexity
Boundaries
Dynamics
Data
Time
Complexity
Wicked
problems
Structured
problems
Tangled
problems
Boundaries
What goes in the model, what is left out, and why
and how do the choices matter?
• Organizations, jurisdictions, sectors
• Groups, individuals and their interests
• Existing policies and laws
• Contextual factors
• Processes
• Practices
• Geography
• Behaviors
Dynamics
Feedback structure of URBAN1, Ghaffarzadegan, et al.
Data
The “best available data” (BAD)
is inherently flawed
Knowledge problems
Data problems
•
•
•
•
•
•
•
•
•
•
•
Meaning
Explicitness
Codifiability
Embeddedness
Dynamics
Accuracy
Completeness
Bias
Timeliness
Availability
...
Time
Problem definition
Goal setting
Policy formulation
Strategy formulation
Deployment
Implementation
Operations
Design
Development
Performance assessment
Evaluation
Policy modeling over time
Problem definition
Goal setting
Policy formulation
Strategy formulation
Deployment
Implementation
Operations
Design
Development
Understanding
the problem
and the options
Exploring
alternative
approaches
Performance assessment
Evaluation
Understanding
the policy
system
Evaluating and
interpreting
policy impacts
Enter stakeholders . . .
Stakeholders bring
• Different knowledge and points of view
• Different goals and preferences
• Different capabilities and authority to act
• Different cultures (national, organizational,
professional)
• Different amounts and kinds of power and
autonomy
• Different levels of confidence and trust
• Different amounts of tolerance for risk
An example: eradication of polio
with thanks to Dr. Kimberly Thompson & colleagues
Collaborators
The collaborators gratefully acknowledge financial support in the form of unrestricted gifts to the
Harvard Kids Risk Project and grants from the US Centers for Disease Control and Prevention (CDC):
U50/CCU300860, U01 IP000029, NVPO N37 (FY2005), 200-2010-M-33379, 200-2010-M-33679,
200-2010-M-35172, U66 IP000169, the World Health Organization (WHO) APW200179134, and the
Bill & Melinda Gates Foundation: 4533-17492 , 4533-18487, 4533-21031. I thank Radboud Duintjer
Tebbens for embarking and continuing on the entire journey with me. I also thank collaborators
from the CDC, including James Alexander, Lorraine Alexander, Larry Anderson, Gregory Armstrong,
Albert Barskey, Brenton Burkholder, Cara Burns, Victor Cáceres, Jason Cecil, Susan Chu, Steve Cochi,
Kathleen Gallagher, Howard Gary, John Glasser, Steve Hadler, Karen Hennessey, Hamid Jafari, Julie
Jenks, Denise Johnson, Bob Keegan, Olen Kew, Nino Khetsuriani, Robb Linkins, Naile Malakmadze,
Rebecca Martin, Eric Mast, Steve McLaughlin, Steve Oberste, Mark Pallansch, Becky Prevots,
Hardeep Sandhu, Nalinee Sangrujee, Jean Smith, Philip Smith, Peter Strebel, Linda Venczel, Gregory
Wallace, Steve Wassilak, Margie Watkins, and Bruce Weniger, and from the WHO, including Bruce
Aylward, Fred Caillette, Claire Chauvin, Philippe Duclos, Esther deGourville, Hans Everts, Marta
Gacic-Dobo, Tracey Goodman, Ulla Griffiths, David Heymann, Scott Lambert, Asta Lim, Jennifer
Linkins, Patrick Lydon, Chris Maher, Linda Muller, Roland Sutter, Rudi Tangermann, Chris Wolff, and
David Wood. I also thank the Global Polio Laboratory Network, Harrie van der Avoort, Francois
Bompart, Anthony Burton, Konstantin Chumakov, Laurent Coudeville, Walter Dowdle, Paul Fine,
Michael Galway, Shanelle Hall, Neal Halsey, Tapani Hovi, Kun Hu, Dominika Kalkowska, Samuel Katz,
Jong-Hoon Kim, Tracy Lieu, Marc Lipsitch, Anton van Loon, Apoorva Mallya, Phil Minor, John
Modlin, Van Hung Nguyen, Walter Orenstein, Carol Pandak, Peter Patriarca, Christina Pedreira,
Stanley Plotkin, Hazhir Rahmandad, Robert Scott, John Sever, Thomas Sorensen, John Sterman,
Robert Weibel, Jay Wenger, and Peter Wright.
Selected academic results
Thompson, K. M. (2012). The role of risk analysis in polio eradication: Modeling possibilities,
probabilities, and outcomes to inform choices. Expert Review of Vaccines, 11(1), 5-7.
Thompson, K. M. (2006). Poliomyelitis and the role of risk analysis in global infectious disease
policy and management. Risk Analysis, 26(6), 1419-1421.
Thompson, K. M., & Duintjer Tebbens, R. J. (2012). Current polio global eradication and control
policy options: Perspectives from modeling and prerequisites for OPV cessation. Expert
Review of Vaccines, 11(4), 449-459.
Thompson, K. M., & Duintjer Tebbens, R. J. (2011). Challenges related to the economic
evaluation of the direct and indirect benefits and the costs of disease elimination and
eradication efforts. In S. L. Cochi & W. R. Dowdle (Eds.), Disease Eradication in the 21st
Century: Implications for Global Health. Cambridge, MA: MIT Press.
Thompson, K. M., & Duintjer Tebbens, R. J. (2008a). The case for cooperation in managing and
maintaining the end of poliomyelitis: Stockpile needs and coordinated OPV cessation. The
Medscape Journal of Medicine, 10(8),190. Retrieved from
http://www.medscape.com/viewarticle/578396.
Thompson, K. M., & Duintjer Tebbens, R. J. (2008b). Using system dynamics to develop policies
that matter: Global management of poliomyelitis and beyond. System Dynamics Review, 24(4),
433-449.
Thompson, K. M., & Duintjer Tebbens, R. J. (2007). Eradication versus control for poliomyelitis:
An economic analysis. The Lancet, 369(9570),1363-71.
Thompson, K. M., & Duintjer Tebbens, R. J. (2006). Retrospective cost-effectiveness analyses
for polio vaccination in the United States. Risk Analysis, 26(6),1423-1440.
Plain language communication of results
Some suggestions
• Build relationships with policy makers and policy
advisors
• Treat models as learning objects to be improved by
interaction with the people who own the policy problem.
• Adopt techniques that engage stakeholders
• group model building, participatory design,
value-sensitive design
• Learn and speak the language of practice
• Custom-make presentations for policy makers and
managers
• Consider multiple methods to understand context
and complexity
• case studies, test beds, evaluation studies
• Update, iterate, adjust, repeat
Many questions, many
answers, better models
The big policy question
÷
Elements of complexity
Boundary questions
Data challenges
Dynamic interactions
Policy
Time considerations
modelers
X
Multiple iterations
Plain language
=
Insights, better understanding of
policy options and effects
Policy makers
Other
analysts
Policy stakeholders
To continue the discussion
•
•
•
•
•
•
eGovPoliNet project, http://www.policy-community.eu/
Policy Making 2.0 community on Linked In
PIN-l listserv
PIN website: pin.asu.edu
Handbook of Policy Informatics (forthcoming, MIT Press)
International Digital Government Research Conference (dgo)
• Policy Informatics Panel, June 17-20, Quebec City
• International Research Society for Public Mgmt (IRSPM)
• Policy Informatics Track, April 10-12, Prague
• Twitter hashtag: #policyinformatics
www.ctg.albany.edu