R
Project AIR FORCE
Research Brief
Assessing the Validity of Uncertain Models
Military planners and other decisionmakers often use mathematical models to simulate realworld situations for purposes such as strategic planning, needs assessment, and testing new
warfighting capabilities and concepts. These models must be validated to ensure that they are an
adequate representation of the real-world situation for the purposes at hand. Model validation is
relatively straightforward in cases where a model is based on settled theories and reliable data.
However, planners often face circumstances in which the validity of a model or its data is
uncertain, even massively uncertain.
RAND Project AIR FORCE has reported a set of concepts that will improve analysts’ ability to
deal with uncertain models and data. Key principles include the following:
•
Models should be comprehensible and explainable to subject-matter experts. One way
to help validate an uncertain model is to make sure that its structure and findings are
explainable in real-world terms. This goal can be accomplished by developing sound,
credible, coherent, and insightful stories to illustrate a model or findings. Such stories
will explain the logical relations between parts of the model, such as cause-and-effect
mechanisms. Analysts should develop several stories at different levels of detail to
explain the model to both fellow analysts and high-level decisionmakers
•
Models should deal effectively with uncertainty. Analysts typically seek to remove
uncertainty by making assumptions about aspects of a model that are otherwise unknown.
A more valid approach is to incorporate uncertainty within a model. New methods such
as multiresolution, multiperspective modeling (MRMPM) and exploratory analysis can
help accomplish this goal. MRMPM uses a family of models and games to approach
complex problems from different points of view. This method allows analysts to crosscalibrate models that have differing resolutions, sources of data, and perspectives.
Exploratory analysis exercises a model across a range of circumstances and assumptions
that represent relevant uncertainties. Such analysis treats the uncertainties simulateously
rather than one at a time. This approach allows analysts to assess the robustness of
conclusions to uncertain inputs.
•
The concept of validation should be broadened to include appropriate qualifiers. Some
models are valid for limited applications even if they cannot be applied reliably under all
conditions. Under current criteria, these models would not be recognized as valid.
Analysts should be able to use qualifiers such as “valid for exploratory analysis” or
“valid, subject to the principal assumptions underlying the model, for exploratory
analysis” when characterizing a model’s validity.
This research brief summarizes the findings of Project AIR FORCE work that is fully described in
Implications for Model Validation of Multiresolution, Multiperspective Modeling (MRMPM) and Exploratory
Analysis, James H. Bigelow and Paul K. Davis, RAND, MR-1750-AF, 2003, 85 pp., ISBN 0-8330-3482-0.
Copies of this research brief and the complete report on which it is based are available from RAND
Distribution Services (Telephone: 310-451-7002, toll free 877-584-8642; FAX: 310-451-6915; or email:
order@rand.org). Abstracts of RAND documents may be viewed at www.rand.org. Publications are
®
distributed to the trade by NBN. RAND is a registered trademark.
The RAND Corporation is a nonprofit research organization providing objective analysis and effective
solutions that address the challenges facing the public and private sectors around the world. Its
publications do not necessarily reflect the opinions or policies of its research sponsors.
1700 Main Street, P.O. Box 2138, Santa Monica, California 90407-2138 * Telephone 310-393-0411 * FAX 310-393-4818
1200 South Hayes Street, Arlington, Virginia 22202-5050 * Telephone 703-413-1100 * FAX 703-413-8111
201 North Craig Street, Suite 202, Pittsburgh, Pennsylvania 15213-1516 * Telephone 412-683-2300 * FAX 412-683-2800
RB-103-AF (2003)