LIBRARY
OF THE
MASSACHUSETTS INSTITUTE
OF TECHNOLOGY
-^hSS. INST.
7?^
MAY 10
1971
t./BRARlE*
.M414
71
C.2
WORKING PAPER
ALFRED
P.
SLOAN SCHOOL OF MANAGEMENT
.Research Program on the Management of.
Science and Technology
The
Delphi
Technique:
Survey and Comment
Alan R. Fusfeld*
# 520-71
-
March, 1971
massachusetts
institute of technology
50 memorial drive
^W :bridge, massachusett
...
..
G
1971
D£W£Y LI3RARY
Research Program on the Management of
Science and Technology
The
Delphi
Technique: Survey and Comment
Alan R. Fusfeld*
# 520-71
"'
March, 1971
* Alan R. Fusfeld is a doctoral candidate at the Alfred P. Sloan
of Management, The Massachusetts Institute of Technology, Cambridge,
Massachusetts.
no
C.J?
RECE'VFO
MAY
M.
I.
I.
4
1971
Llilh(Afi!lES
7/
THE DELPHI TECHNIQUE:
by
Alan
R.
SURVEY AND COMMENT
Fusfeld
Even though DELPHI has been used with some success, ..., it should
not be interpreted as a device that produces 'truth about the future'.
in 'inexact'
The Delphi method is designed to produce consensus judgments
it would be a mistake to consider such judgments as complete or
fields;
precise descriptions about the future.
Theodore J. Gordon and Robert Ament,
"Forecasts of Some Technological and
Scientific Developments and their
Societal Consequences", IFF R-6,
September, 1969
Introduction
:
Delphi forecasting is a widely used technique for the systematic development of expert opinion consensus concerning the future
''1).
This article
considers the implications for Delphi of recent analyses of forecast results
and of the historical development of the Delphi methodology.
It
is
important to note at the outset that the theoretical utility of
Delphi is that it pools individual opinions in order to develop forecasts which
are more accurate than those of any average individual.
This has been the
major driving force behind its development.
The potential utility of Delphi, described above, was recognized by Norman
Dalkey and Olaf Helmer in 1951-1952,
They were working for the RAND Corporation
and applied Delphi methodology to Air Force strategy problems in Project
"DELPHI" (2).
Their work, however, was not oriented towards the timing or
nature of future events, as we speak of these characteristics today.
Rather, they and other early developers were interested in
-
'A
-
.
.
policy formation research, operations research, and political gaming
Some
of these
early uses were directed
schedules for U.S. locations,
election outcomes; and
case
in the
of
war on
July 1,1953;
made
(2)
predicting
Communist China.
miethodology through "almanac" delphi
to validate the
forecasts, which are predictions
).
estimating Soviet bombing
(1)
predicting U.S. policy towards
(3)
Attempts were also made
at:
{3
for questions with obscure but
known
answe rs
This early work was extended
conjunction with Olaf Helmer,
in
1963-64
,
by Ted Gordon working in
include the long range forecasts, which
to
Since then, Delphi studies have been conducted in such
are now familiar.
diverse fields as medicine, education, space hardware, oceanography, and
The spcns'^ring organizations have also been widely varied-
transportation.*
government agencies (U.S. and foreign), RAND, IFF, Goodyear Tire and
Rubber Corp.,
^.
MA
TRW
Py 9^ fo, rt^
Corp.
/2y /jyy,
,
LTV
/^
ft*,
brief consideration of the
Corp.
17,
,
McGraw
JJ
IS
summary
-Hill Corp.
,
al.
(^^
)
description of the development of
Delphi, just presented, prompts several interesting questions.
which are most important, because
et.
of their vital
The two
positions in the rationale of
Delphi methodology are:
how valid are the "almjanac" delphi forecasts?
(2) how similar are long range delphi forecasts to
(1)
the processes of
estimation and short-term prediction, for which Delphi was first
devised ?
An understa^Xiding
Delphi forecasts
of these
in
The remainder
to
questions
is
essential to the efficient utilization of
corporate planning.
of this article is
structured into the following five sections
help provide th^underst^ding:
(1)
background and basic rationales for
the
development
of
Delphi
forecasting;
Delphi methodology;
real-world uses of Delphi (includes cases, which illustrate uses of
material presented in the preceding sections and provides a focus
for questions concerning the behavior of Delphi);
recent studies and implications of forecast results; and
corporate use of Delphi (draws preceding material together to suggest
how corporations miight increase their benefits fromj Delphi, through
an improved understanding of its nature).
(2) extensio;-."s to
(3)
(4)
(5)
Institute for the Future (IFF), Riverview Center, Middletown, Conn.
perhaps, has the greatest amount of available information regarding current
Delphi studies (over 2000). Olaf Helmer is president of IFF and Ted
Gordon is a senior fellow.
*The
,
.
-3-
Background and Rationales
The background
of
Methodology
:
Delphi methodology extends into the philosophy of
of
the solicitation of experts' opinions.
Norman
Rescher
Dall<^ and Nicholas
give a fine description of this and contend that there are certain fields of
knowledge, which, by their nature, commonly require the opinions
expert
(«IC^.
answers are
These
not
fields are
termed "inexact," meaning
of an
that absolute
known or possible, and include such areas
as juris-
prudence, political policy formation and corporate strategy.*
The opinio
whe«'&
making,
ns of experts, then, are justified as inputs to decision
Of course, expert judgment
judgment-in an "inexact" area.
-
has been sought by decision makers for thousands of years, but, unfortunately
many of the experts,
Some striking examples
for
of future
of
were
misjudg nsents by experts,
bow
is
many ways.
bowman can let off
Whereas also
a
musketeer can discharge but one
a
Privy Council
(2) "
in the
in
.
.
six
aimed shots a minute,
two minutes.
"
(AdviSe to the British
by Colonel Sir John Smyth.) (A/)
no plea which will justify the use of high-tension and
in 1591
There
employed solely
is
to
reduce investment
in
commercial sense.
November,
1889)
"...
that
greatness as a scientist
is
r
om
Professor Langley will not put his substantial
in further peril
by continuing
to
involved, in further airship experiments.
waste his time,
Life is short, and
trying to fly. ...
"
(
New York Times
the first flight of the
editorial,
Wright brothers)
December
10,
1903
.
U-f).
of "inexact" fields would be "exact" or precise areas of
knowledge. Examples of "exact" sciences are physics, mathematics,
and chemistry. It should be clear that many areas, particularly the
social sciences, combine "inexact" and "exact" sub-fields.
The opposite
*-!'
,
capable i« se-rvices to humanity greater than can be expected to result
week before
-
American Review
OT
he
f
"
(2').
We hope
money
They are
copper wire and real estate.
(Edison, T.A., "The Dangers of Electric Lighting," North
and the
"inexact" area
and tires out soldiers on the
alternating currents, either in a scientific or a
(3)
''*
.
weapon, firearms are very complicated
a simple
things which get out of order in
march.
not always correct
technology, appear below:
"... The
(1)
the judlgments
The reader, wh^is not already familiar with the origins of the word
"DELPHI," is reminded that it refers :o the oracle 'of Apollo located
at Delphi, a place in ancient Greece {ZA)
.
.
one
A-
may
the fact that single experts
Given
hold incorrect opinions,
is
it
logical to seek increased accuracy through the opinions of a collection
This assumes, though, that a representative statistic of the
experts' opinions may be chosen, which will be more accurate than the opinion
individual
of the average exoert, i.e. it will somehow be corrected for
experts.
of
bias and misinformation.
One
formal attempts
of the first
to
systematically aggregate the
DELPHI," begun at RAND Corporation
The work was conducted by Norman Dalkey and Olaf Helmer,
opinions of experts was "Project
in I95I-I952.
who hoped
to
develop a methodology for comibining the opinions
on the subject of estimating Soviet offensive strategies
Dalkey and Helmer
built their study on earlier
public opinion research.
='•
A
(
work
2»
of
experts
)
area
in the
of
1948 study in this area, "The Prediction
of
Social and Technological Events," by Kaplan, Skogstad, and Girshick,
is
particularly interesting, since
(a)
it:
which "Project DELPHI" developed and
illustrates the background in
(b)
presents evidence that
statistical aggregates of individuals' opinions, for near termi events, are
more accurate
individuals
than predictors derived from group interaction ot single
(^
).
The Delphi technique was formed by adding
information feedback and iterations
of the
concepts
to the
of
controlled
aggregation of
some psychological
technique eliminated
(e.g. fear of superiors, fear of changing opinion) by keeping
effects of groups
individuals
responses
An additional aspect
individual opinion.
tje
of
the
anonymous
RAND
estimates for the
to
each other.
Delphi methodology provided
policy/strategy studies, that should have been
more
accurate than those of other types of groups or of individuals for two
reasons:
and
(2)
(1)
the
avoidance
the increase in
of
psychological biases
accuracy from a pooling
Following the developmient
towards experiments designed
ments applied
Idaho in I960?).
(2)
is
It
(e.g.
to test its validity
How man
The responses
to
committees, panels, etc.;
expert judgment.
Delphi technique, attention was turned
the Delphi msethodology to questions
known but obscure
(1)
of the
of
of
(0,*1^,**
where
y tons of potatoes
) ,
These experi-
the
were harvested
in
such almanac questions were shown
converge during several iterations;and
converge towards the correct answer for the majority
the "almianac" Delphi results
answers were
which were and are used
of the
to:
questions
.
to verify the
usefulness of Delphi.
*
The major journal
in
=;==:=
of this field is the P"'^blic
Opinion Quarterly
January, 1937.
The analysis
is
discussed
in detail in a later section.
,
established
''*
As indicated earlier,
to
methodology was not generally applied
the Delphi
questions concerning the future until 1963-64.
which influenced
this.
the "Project Delphi"
First was the release for publication of some of
work
in 1962,
The subject matter
(
required classification for security reasons.)
growing interest
Several things happened
time
at this
in
of the
work has
there
In addition,
was
a
forecasting technological developments,
i.e.
an increased awareness of the impacts of technological change on long-range
plans and an increased
demand
(e.g. PPBS). Finally, in 1963,
for quantification in all aspects of planning
Ted Gordon
Director
(
of
Space Station
and Planetary Systems for Douglas Aircraft Company) met Olaf Helmer
by chance, during a luncheom meeting at
RAND
^
(
point that Gordon's interests in technology planning
Helmer's interests
this
in the utiiization of
(
was
It
were linked
expert opinion.
was the application of the Delphi technique
The Helmer-Gordon work was the first use
(greater than one year) forecasting
),
A
to
consequence
to future -oriented
of
of
questions.
Delphi for long-range
Their study ("Report on a
/7).
Long -Range Forecasting Study") concentrated on six areas
(1)
at this
of interest:
scientific breakthroughs; (2)automation; (3) space; (4)population;
war prevention; and (6) weapon systems. These were different in two
respects from the topics considered in "Project Delphi." First, HelmerGordon topics required an estimation of all of the many interacting dynamics
actors influencing technological advance, whereas "Project Delphi" topics
(5)
of
required an estimation of strategies based on role playing, with technology and
other variables fixed or in a static state.
of the
topics
unlimite
c'l
Also, the relative time horizons
i.e. a year or less for "Project Delphi"
were different,
time for the long-range study.
(
These differences are
important for the comparison and evaluation
of
future oriented studies,
"almanac" studies, and "Project Delphi" studies, discussed
later section.)
It
two applications
is
of
the beginning of the
clear
,
and
in detail
in a
though, that regardless of differences between the
Delphi methodology, the Helmer-Gordon work marked
use
of the
Delphi technique for technological forecasting.
Delphi methodology, then, became a technique for forecasting technology,
through the linking of quantitative planning needs with the procedures developed
by Project Delphi."
ground
(
S
(f
fO
"2.
The theory and rationales also come from
2.0^3
33
Zi, /"J
).
this
back-
They may be summarized by
the following structure, consisting of a
premises, and a group
major premise, several nminor
of postulates:
Major Premise:
Future events occur due to the complex
interactions of nnany ^rces.
Minor Premises:
1.
Many elements involved
of future
in the projection
developments are not easily
quantifiable.
2.
non-exact disciplines, expert opinion
"modeling" must, of necessity, substitute for the exact laws of causality found
In
in the physical sciences.
3.
The prediction of outcomes or occurrences
of future events, with any degree of probability, is an "inexact" science, just as
jurisprudence or organizational strategy
4.
is.
Individual experts suffer biases and are
incorrect nearly as often as they are correct in predicting future events.
Postulates:
1.
It is justifiable to solicit and bring to
bear expert knowledge and judgment concerning future developments.
/
2.
of experts have aggregate-relevant
information that is equal to or greater than
any member of the group.
3.
Groups of experts have aggregate misinformation that is equal to or less than any
Groups
member.
4.
5.
Groups of experts have at least as many
informal models (systematic processes
for analyzing information), by which an
answer is estimated, as any member.
Groups which meet face-to-face,
in order
develop a consensus, suffer bias from
follow the
psychological pressures (e. g.
leader syndrome or reluctance to change
previously- state opinion).
to
,
6.
Groups whose members are anonymous to
each other, develop a consensus which
corrects for an individual bias and removes
the possibility of a psychologically-developed bias.
7.
The median response of a group is at least
as good as one half of the respondents (by
definition).
Together, the postulates imply that the application of the typical Delphi
technique, consisting of
a)
anonymity
b)
controlled feedback
c)
an estimator of group opinion^ will provide a jnore
accurate response to a question than that of the
average expert.
A
is
of experts
-
-
to eliminate bias,
to
develop consensus, and
description of the typical Delphi process appears below and
accompanied
in
Figure
1
by several examples of Delphi forecasts.
should be apparent that the actual process
fits in
It
well with the preceding
discussion of the underlying premises.
A
typical [forecast] is initiated by a questionnaire
of a set of numerical
quantities, e. g. dates at which technological possibilities will be realized, or probabilities of
realization by given dates, levels of performance,
and the like. The results of the first round will
be summarized, e. g. as the median and interquartile range of the responses, and fed back
with a request to revise the first estimates where
appropriate. On succeeding rounds, those individuals whose answers deviate markedly from the
median (e. g. outside the inte r- qua r tile range)
are requested to justify their estimates. These
justifications are summarized, fed back, and
which requests estimates
,
The counter agreecovinter -agreements elicited.
ments are in turn fed back and additional reappraisals collected.L
/
[Preliminary questionnaires are often used to
select and develop questions for which estimates
will be collected.
In
as a
new
summary,
the development of the Delphi technique to 1964^
tool for forecasting^followed a logical sequence of steps.
is illustrated in
the
]
Figure
2
which also serves
development which should be
to highlight
two aspects
of particular interest to
This
of
present and
:
Figure
.
I
EXAMPLES OF DELPHI
Description of Event
FORECASTS
Date By Which
25
50
Reference
75 %
of Respondents
Think Event
Occur
Wi
I
I
Year
Forecast Made
There will be a single
netional bui ding
code.
1975
1977
1980
1968
Polymers will be
created by molecular
tai loring,wi th
service temperature ranges in
excess of 1000 F.
1971
1976
2000
1968
1980
1982
1985
968
1974
1983
1990
967
1990
2015
2015
969
I
SST
ai re raft will
be in regular
scr V ce over
land areas,
1
Hydrocarbon/ai
fuel
eel
I
r
will
be commercia
avai able
I
ly
I
A source of transplant
organs for humans
will be developed
through selecMve
breeding of animals
that are tissue
compatible.
27
potential users of Delphi (see dashed lines in Figure
assumption that forecasting
tacit
DELPHI
involved in Project
similar
is
to the
The
2).
processes
first is the
estimation
of
and the judgmental processes that are justiThis
fied by the philosophy of the inexact sciences.
is
not intuitively ob-
vious and nnay result in overjustifying the use of Delphi for future tech-
may
nological developments, which, in turn,
create a sense of overcon-
The other aspect
fidence in the forecast results.
of
importance
is the
establishment of scientific objectivity for Delphi forecasts through "al-
manac"
Given that almanac studies do, in
studies.
of Delphi for
almanac
studies
it is
again not intuitively obvious that
(A discussion of the impli-
they are adequate tests for forecast studies.
cations of
almanac
justify the use
fact,
studies results is in a later section.
that users of Delphi forecasts be
aware
of these tacit
It is
)
suggested
assumptions and
their implications and should, perhaps, exercise caution in placing too
much
Z.
confidence in the results.
Extensions to Delphi Methodology
After 1964, several additions were
None
of
crease
them altered
its utility
(3)
(2)
to the
SEER
These extensions consist
discussed briefly below and, of course,
is
of (1)
cross-
(System for Event Evaltiation and Review);
integration of trend extrapolation with Delphi>>
sumption that Delphi
Delphi technique.
basic design, but rather they were aids to in-
or efficiency.
impact analysis;
and
its
made
an acceptable,
if
is
'
'
'
.
Each
is
based on the underlying pre-
not perfectly accurate, fore-
casting technique.
Cross impact was developed
to
tionships between forecasted events (28,
which consist
solve the problem of inter-relaIn other
29).
words, forecasts
occurrences, likely dates
of collections of potential future
occurrence and their probability might contain mutually reinforcing or mutually exclusive events. An example of a reinforcing interrelationship would be the discovery of the means for the magnetic conof their
tainment of magneto -hydrodynamic plasmas
development
is illustrated
the
of
positively affecting the
commercial thermo-nuclear power.
by the
development
of
feffect of
A
negative impact
continued inexpensive source^ of coal on
economically viable nuclear power.
Basically, the
technique consists of establishing a matrix of the cross -effects of the
various items in the forecast, such that a consistent set of estimates
can eventually be developed.
The
evaluatio'h of cross-effects is the
impact analysis.
and
may
Usually
be in the
form
key variable in cross-
cross -effects are determined by consensus
of a
revised conditional probability estimate for
an event or in the form of a general type of influence that the occurrence
upon the probability of occurrence
of one event has
tency of forecast results
is
complete matrix
is
this
Consis-
achieved through a series of iterations which
revise the evaluations of the cross -impact matrix.
assumptions are made,
of another. *
When
can be done by computer.
simplifying
Eventually, a
developed which indicates modes of linkage between
events, strengths of relationships, and predecessor-successor orientations.
Thus, cross -impact analysis
user
of Delphi,
since
it
an important technique for the
is
eliminates contradiction and derives an internally
consistent forecast of revised probabilities of events, at least
some
of
which are interrelated.
Another addition
SEER
to Delphi is
uation and Review(4). This technique helps to
more
efficient
process by developing
views, thereby avoiding a flaw of
--
System
for Event Eval-
make Delphi
initial lists of events
many
forecasting a
through inter-
forecasts which start with a blank
page and by clearly indicating a set of rules, which reduces the amoiont
of
time required by the Delphi "panelist".
The time needed by the fore-
cast process can be quite considerable, particularly when hundreds of
events are involved.
SEER
limits the time investment by constraining
the forecast to two rounds and requesting that panelists
those questions within their areas of expertise.
the question of "event desirability"
feasibility"
from
from
SEER
answer only
also addresses
a user's point of
a producer's point of view.
view and "event
Finally, utilization of this
technique produces a Delphi forecast in which interrelationships, goals,
supporting events, and alternative paths are clearly specified.
-•''-!=
* Examples of general types of influence would be enhancing (enabling or
provoking) and inhibiting (denigrating or antagonistic) [28].
** The interested reader is advised to see: "SEER: A Delphic Approach
Applied to Information Processing", by G. B. Bernstein and Marvin
Cetron, in Technological Forecasting VI, N. 1, J\ine 1969.
,
~
The third extension
to the
I
V
Delphi technique
is
an integration of
cross-impact analysis, SEER, and trend extrapolation with the Delphi
process
The new factor
(13, 16).
in this
approach
is
the application of
Panels are given the his-
Delphi to the deternnination of future trends.
toric (time series) data and asked questions designed to elicit their
opinions concerning the future
e^ironment and direction
design and results of this process are similar to
matrices.
to
It is
a significant addition to Delphi since
their intuitive
rfelate
SEER
models
it
of the future to actual,
The
of trends.
and cross-impact
forces the experts
as opposed to
perceived, historical trends.
3.
Case Applications
At this point,
it is
Delphi forecasts have been
useful to consider brief discussions of
made and
utilized
how
by four different organi-
They indicate the range of complexity that has been attempted
and include Ling-Temco- Vaught Corporation, TRW, Institute for the
zations.
Future, and Goodyear Tire and Rubber Corporation.
LTV
initiated technological forecasting (basically in
Delphi) in 1968
(8).
terms
of
Their forecasts consisted of approximately one
hundred events and were incorporated into corporate plans through inclusion in their
R&D
planning cycles.
grounds were used
to
Panelists with non-technical back-
broaden the base
of intuitive
models and thus
in-
crease the awareness of the Delphi group to non-technical aspects of
the questions, e. g. economic, marketing, or environmerftal factors.
In addition,
the feedback for iterations included the percent of "never"
responses in order
ity
to
which were observed in certain
"never's" feedback.
casts were
LTV
Finally,
it
made with estimates
LTV
tests without the percent of
should be noted that the resulting foreof probability
and indications
of
which
divisions would be primarily effected by each event.
Complex but
of
preclude the possibility of trends toward conform-
TRW's
straight Delphi forecasts are also the
forecasting efforts which began in 1966-67 (6,7).
trademark
Their
studies have covered diverse fields -- space, military, transportation,
housing, communication/education, materials, power, and biology/
oceanography and contained hundreds
of events.
TRW
integrated their
forecast information through the use of future "mapping" in order to
enhance the
utility of
such diverse fields of information for corporate
planning.
A
Donald
Pyke, Manager, Technical Liaison and Forecasting,
L.
succinct description
TRW's mapping
given below by
is
TRW,
Inc.
:
[Mapping is] a concept for a comprehensive system
which utilizes available forecasting techniques to generate a map of the technological future which will provide for the planner or assessor a tool similar to that
.
.
.
available to the travelers. As envisioned, this map
would display technological alternatives and their environmental consequences in such a way as to enable
the coupling of near term technical activities to long
For maps of the future, one axis of
range forecasts.
coordinate
system
is time and the other is the catethe
gory under investigation. An hierarchical system is
envisioned in which the summary level of the environment subdivided into the eight aspects identified.
[as] technological, commercial, political, social,
ethical, cultural, personal and ecological.
(7)
.
.
.
.
It is
clear that
TRW
.
.
has designed a sophisticated vehicle for
use of
its
Delphi forecasts, that, perhaps in simpler form, should interest
many
others.
An organization
for the Future, which
studies.
is
one of their
In
that contrasts with
LTV
and
TRW
the Institute
is
concerned primarily with contract forecasting
many
Delphi studies, they combined traditional
event forecasting with Delphi forecasts of trends, panels for role
playing, and cross impact analysis (13,14,15).
sophia^ed use of Delphi forecasting.
It is
note the use of panels for role playing.
This
is
probably the most
particularly interesting to
These were employed
to
simulate
the attitudes of various sections of society at various times in the future,
given alternative sets of 'facts'.
Panels, such as these, might also
be very useful for corporate planners.
In
order that the reader not think that
all
Delphi studies must be
as complex and sophisticated as those discussed above, the following
application by Goodyear is presented
First, a group of experts
sinaple steps.
which influence the
tire industry
were
(30), *
from
They went through three
the principal organizations
were requested by mail
most important developments they
2000"
(9, 30).
anticipate between
".
.
.
now and
to list the
the year
Next, a relatively few event questions of particular importance
selected.
Several questionnaire iterations of the revised
list
*The organizations included tire companies, fiber companies, chemical
technical
companies, carbon block companies, industry consultants and
,
trade journals.
concludeiAhe study.
is
It
important
to note, that this study,
which
less
is
complicated than the others mentioned earlier, was judged as very useful
to
Goodyear's research plans.
The reader
probably aware that the forecasts just discussed are
is
based upon an acceptance of the accuracy of Delphi.
However,
this is a tacit
assumption that has not been clearly explored, either, by the rationale
methodology or by empirical evidence because
of Delphi
nature of the forecasts.
planner who intends
intuitive
things:
to
of the long range
Thus, until confirming evidence
is
available, the
use the Delphi process as a method for aggregating
judgements concerning the future, must, himself, decide two
(a)
how accurate does he believe Delphi forecasting
and
is?
(b) to
what extent can Delphi forecasting be relied upon- -for what length of time?
If
even relatively simple techniques
this is not done,
may
lead him, unaware,
into future difficulty.
Behavior and Analysis of Delphi
4.
In the last several years, a
on Delphi
35).
'arf<so.r\a.c
An analysis
'
number
of studies
have been conducted
data and Delphi forecast data (24, 25, 31, 32, 33, 34,
of these studies provides additional information for
understanding the behavior of the Delphi process.
A summary
of this
analysis and the implication for the use of Delphi are presented below in
the following
framework:
1.
behavior of Delphi responses for 'almanac' questions;
2.
behavior of Delphi responses for future questions;
3.
summary
4.
other comments.
The reader
to
will recall that Delphi 'almanac' studies consist of
questions with known, but obscure answers.
questions are:
"--how many million board
in the United States in
the United States
and
of analysis of results of the 1964 Delphi forecast;
1962?
farmer for
Some
feet of
responses
actual sample
lumber were produced
--what was the average price received by
a bushel of apples in 1940?
--what
is the
specified operational gross weight, in pounds, of the Gemini capsule
(exclusive of occupants)?
"
(23).
Such questions have been given
business students, and
members
to
RAND
research
of industrial corporations.
staff,
graduate
The results
have been quite similar and yield a number of interesting conclusions.
With regard
interest
accuracy of responses, a primary item of
average group error decreases with
is that
group size,
to the
increases in self-rating, and
(b)
(c)
increases in
(a)
decreases in the amount
among the respondents. This can be seen in Figures 3 and
4 quite clearly, A note should be made here that while dispersons and selfrating each correlate well with error when the other variable and group
of dispersions
size (Figure
are held constant, there
3)
independence of group size (Figure
held constant,
it is
is
no data to substantiate the
In other
4).
words, with self- rating
possible that the size of the group
group error by decreasing the dispersion (dispersion
group size) or the error
may decrease
may decrease
is
the
dependent on
with increased size for any given
dispersion (dispersion independent of group size).
of interest to note the effects of dispersion,
Regardless,
it is still
self-rating, and, possibly,
These effects would appear
group size on the accuracy of response.
to
be useful in designing and evaluating Delphi studies.
Another area of importance
is
reproducibility of results.
in relation to the design of a Delphi
The 'almanac' studies indicate
(medians) from different groups of the same size became
alike as the
number
in the
groips increases.
that results
more nearly
The correlations between
pairs of groups of various sizes are exhibited in Figure
5.
This lends
additional weight to the significance of gro\p size, mentioned before with
reference
to
accuracy, as a Delphi design variable.
The characteristics
of 'almanac' studies discussed
above are in
reference relative performance (accuracy, reproducibility) of different
types of groups (differences in size, self- rating, and dispersion).
are important for the design of a Delphi study, but in order
utility of the
to the
Delphi
it is
necessary
performance behavior of
to go
to
These
maximize
the
beyond relative performance
a f^AaiUS group.
There are three aspects
to this:
1.
the lognomal distribution of individual responses within
a group;
2.
the lognomal distribution of the
group about the true answers for
3.
the
normal distribution
responses
* Self-ranking
to
median responses
of the
a series of questions,
of the relative
and
accuracy of median
those of an individual in the group.
rgfers to the individual respondent's indication of the degree of
his familiarity with the subject material of a given question.
e 3:
GROUP ERROR AS A FUNCTION OF
DISPERSION OF RESPONSES (CD
AND GROUP SELF-RATING
\Rotind
1
Grou|)\
self-rntirin^
ure 4
GROUP ERROR AS A FUNCTION OF
NUMBER IN GROUP
0.3
(24)
0.2
0.1
_L
9
17
13
Number
in
group
21
25
29
JURE
REPRODUCIBILITY
p=mean coefficient of
correlation between
pairs of results of
groups of size N
OF
RESULTS
«
£
7
5
g
s
0-4
3
2
1
5
7.
Number
It is clear
that there
is
a definite
in
U
»
jroup-
(_
^4^
and monotonic increase
of the group responses with increasing group size. It
relationship would be approximately linear between n
is
=
in the reliability
not clear why the
3
and n
=
11.
•'"'
724)
The lognornal distribution
Figure
in
It is
6.
of the individual responses is illustrated
interesting because
it
indicates that the subjects in
'almanac' experiments develop their responses as ration
This means, for example, that individuals select their
magnitudes.
answers from numbers such as
and 150.
or relative
also
It
means
are distributed about
of responses
not be an equal
implies that
if
that if the correct
10 and 100 as
answer
is
opposed
to 50,
100,
100 and the responses
fashion, there will be an equal
between 100 and 1000.
number between 50 and
number
(There will
100 and 100 and 150.
This
)
individual answers fall equally to one side or another of
the correct answer, the
An
random
in a
it
between
100, and 1000, as
10,
median
will be a better estimation than the
mean.
analysis of median responses to a series of questions with
respect to the correct answers also indicates a lognomal distribution
(see Figure
7).
This means that the distributions of answers of the
entire group are randomly distributed about the correct answers.
Further,
they are distributed as ratios of the correct answers.
is that the
panel, as a whole,
may
be off the right
The implication of
answer by a relative
this
magnitude.
Figure
In addition to this,
7
indicates
(1)
that
most medians
most aggregate estimates by the group) tend towards the correct
answer, (2) some medians may be magnitudes aw^ay from the correct
answer, and (3) the dispersion of the medians tends to get smaller vsdth
(i.
e.
,
more
iterations.
This
since
is significant,
it
should allow the development
of confidence limits for the Delphi technique, as applied to 'almanac'
question.
The 'almanac' Delphi also permits an evaluation
terms
the
of the
number
median versus
Of course, this number can vary from 0%
The distribution
accuracy' category
It
is
For each
of individuals
shown
in
Figure
to
some always
individual,
better, but
100% of the
over the various possible 'median-
8.
would seem that some individuals always tend
the median,
accuracy in
which the median was more accurate can be
of questions for
established.
question.
the individual respondent.
of
to
do worse than
most are more accurate
for about
half the question!. Put another way, the average individual's responses.
:gure
6
DISTRIBUTION
OF
INDIVIDUAL
RESPONSES
PI Obsen/ed
^" Log normal
—
-'-..r.fl-oox)'
XV2»*
"2
10
15
20
Normalised estimate
25
30
(24)
URE
7
DISTRIBUTION OF
QUESTIONS
MEDIANS (M) FROM A SERIES
WITH RESPECT TO THE TRUE
ANSWERS (A)
(note:
OF
curves fitted by eye)
Round
/
,26
>
1
aiSTRIBUTlON OF
CATEGORIES
INDIVIDUALS WITH RESPECT TO
RELATIVE MEDIAN IMPROVEMENT
OF
(note:
curve fitted by eye)
.7.
.6
^
/
medians from
'expert'
subgroup
.4^
medians
from
total group
.3
.2
,1-
Of
100
7„
of questions for which median was more
accurate than individual
Data from ref. 23
if
such a person exists,
as accurate or, perhaps, a shade less accurate
is
than the medians.
This would
seem
to
throw doubt on the usefulness of Delphi.
However, when medians for each 'almanac' question
a self-selected subgroup of individuals, they
are recalculated from
show an average improvement
over the individuals' answers for 75% of the questions, as opposed
previous 50%
(see dashed line in Figure
9).
to the
The improvement demonstrates
the usefulness of self-ratings for 'almanac' questions.
In
summary,
the results of 'almanac' Delphi studies show:
1.
a
dependence of relative accuracy on self- ratings;
2.
a
dependence of relative accuracy, on dispersion
3.
a
probable dependence of relative accuracy on group size;
4.
a
dependence of reproducibility on group size;
5.
the lognomality of individual responses;
6.
the lognomality of
median responses about
of replies;
the true
answers; and
7.
the relative
improvement over individual responses
medians taken from a self-selected subgroup
In addition, the studies
have shown Delphi
is
of
of 'experts'.
an improvement over
face-to-face discussions, that consensus does develop with feedback,
and that there
is
an optimum time (30 seconds) for reply
questions (24, 25, 32).
meters for
analysis
to
'almanac'
Together, these resxilts indicate important para-
the design of a Delphi lalmanac' study and suggest that statistical
may
be used
to
characterize
its
regular distribution behavior.
Recently, studies were completed indicating that results of Delphi
forecasts exhibited similar behavior
The two significant aspects
of this
moved
work were
'almanac' studies (32, 34).
the demonstration of the
and the growth dispersion of the respondents
logncffnality of Delphi forecasts
as the forecasts
to that of the
farther into the future.
Lognomal distributions
of Delphi forecast respondents (see
Figure
indicate that the panelists perceive the future in terms of relative
magnitudes, just as the 'almanac' panelists perceived their answers.
the very best, this suggests that there is
relationship that
is
common
to
At
some underlying behavioral
processes of estimation.
This
is
supported
9)
I
1
v^u
I
c;
^
LOGNORMALITY OF DELPHI FORECASTS
0.01
3
10
5
7
Standardized Deviate
.. Cunuilati\c piiibaljiliix distrilnilion of stancl.irdizccl dcxiatis for90"„
of 4.0 was added to cacli dcviaii- lo niakt- all ri-sulis pusiiiw.
The
daiii arc ploiicd
on
Kjijiioi
iii;il
cotu
(lin.iics. 'I'lir
that the cslimalcs arc in fact ioi^noi inally di-iirihuicd.
the conceptual diliiculiics .^ssocialrd wiiii
arbiliaiy
]i(jsiii\c
pioiiiiiL^
i^ood
likt-liliood cslimalcs.
^
A
value
to a slraighi line iiulicalcs
Ii slioiild i)c
iioicd ilial to avoid
a "/cio" on a loi^aritlimir scale, an
consian' wa'; added lo ckIi of the dc\iaics. Tliis in no \\a\ ad'ects the
iognoi niahiy of 'he plot, and the oii^in.d duta can be recoveicd by subtracting the
constant.
The
ized dcxiales
constant used dc])cndcd on the iani;c of values coveicd by the siandaicl-
and
is
indicated witii each fiyme.
Tliis analysis of c^limates ol)Uniicd duiiuu;
roborates earlier fnichngs
made
thrf:n<5ii
an ailempt
to
prepare a forecast cor-
analysis of responses to almanac-type Cjuestions,
(34)
— \L-
by the findings
A
of several behavioral studies. *
recognition of the
lognormality of the Delphi responses with respect to time
essential to
is
estimating the amount of confidence that should be placed on a forecast
as the time horizon increases and
median forecast dates
may
help to explain the occurrence of
that skip logarithmically into the future
(i.
e.
,
fore-
casts regularly give median dates of individual years nearly to the year
2000 and then begin to skip 2005
-
2010
The other important finding
of respondents and
the dispersion
median data
-
2025
-
2050
later
-
never).
=!=*
relationship between dispersion
is the
of the forecast (33).
grows linearly with the time horizon
depends on the particular panel.
-
It
appears that
at a rate that
This suggests that further attention
should be drawn to factors which influence this rate.
In addition, if
one accepts the argument that Delphi forecast studies are at
all like
Delphi 'almanac' studies, then the increased dispersion should correlate
with increased error.
Hence, caution should be taken when using Delphi
forecasts where the time horizon increases the dispersion beyond an
acceptable level (judged, for the moment, individually).
At this point,
appropriate to briefly mention the results of
it is
an analysis of Gordon's 1964 Delphi forecast
(31).
This study found that
approximately 50% of the events were either accurately forecast or were
forecast for the data by a 1969 forecast.
of environmental changes,
technical agents, were
also showed the influence
It
the 1964 space events, forecasted by
since
made under
the tacit
assumption that funding
continue at 1964's level and /or growth rate.
point of hindsight
we know
that the funding
obvious effects on the forecasts.
that
some forecasts are bound
*The interested reader
is
to
It
is
be in
advised
to
From
the comfortable vantage
environment has changed with
clear from the preceding paragraphs
'left field'
the planner should be
The psychophysics
,
of
aware
examine:
Stevens, S.S., On the psychophysical law, Psychol, Review
Stevens, S.S.
48, 226-252,
would
,
64,
153-181, 1957.
sensory function, American Scientist,
I960.
Stevens, S.S. and E. H. Galanter, Ratio scales and category scales for a
dozen perceptual continua, J. of Experimental Psychology 54, 337-411, 1957.
,
Teichner, W. H.
3-9,
,
Psychophysical concepts
of probability,
Psychol. Reports
,
10,
1962.
of the uncertainty estimat:ion modjels of A. Wade
Essentially, he assumes that individual Delphi respondents
have an underlying lognormal probability distribution relating to the event in
question. I-^e tnen applies Bayesian statistical techniques to revise forecast
*=''Npte
should be
Blackman
made here
(35).
confidence limits.
The 1964 study revisited shows some
of the possible reasons for
the variance in this case and, also, suggests
some improvements, such
of this.
covering
all of the 'bases',
e.g.
,
Perhaps a 1964 Delphi
funding.
should have been conducted with Congress.
this
would have helped, but
it
as
There
is
no guarantee that
would have provided a more complete
perceptual picture.
Other aspects of Delphi that should be considered by the corporate
planner
lie in
What
of questions.
name
afraid to
These are best framed
yet unexplored areas.
is the
in the
psychological bias of the year 2000?
form
(Are people
dates that cross the end of the second millenia?
Does the
year 2000 affect the perception of time, as dates are named that near
What individual characteristics,
What kind
Delphi panelists?
forecast
to a panel,
Is it
)
influence its
life 'bias'
of perceived risk taking is involved in the
a certain range is too
planning?
common
Does the position of an individual in his career or
forecasts?
results?
if
it?
minds of
possible that considering events in the future beyond
'inexact' a science and is not a legitimate part of
Do Delphi panels take
may somehow
influence
full
cognizance of the fact that their
the event?
What are
the confidence limits
that can empirically be placed or justified for Delphi forecasts?
The answers
to
these and other questions raised by the preceding
discussion are, certainly, not available readily.
of their possible effects is
paramount
However, a consideration
to the truly effective
forecasting and they should, therefore, be resolved in
use of Delphi
some form by
the
corporate planner.
5.
Recommendations for Corporate Utilization
of Delphi
This article has presented material that contains various levels of
implication for corporations desiring to apply the Delphi technique to
forecasting.
For purposes
of clarity, the suggestions
discussed belo'w
are limited to improving the basic structure which Delphi
The planner or organization must
first
match
desired with what can be produced from Delphi.
and resolved
at the outset,
with naive and intuitively obvious results,
(c)
employed.
the kind of output
this is not
addressed
well before people are lulled into false security by
sophisticated computer printout, the Delphi user
obviously be delivered,
If
is
(b)
making forecasts
may
find himself (a)
working
promis*»»«jmore than can
that are not used, or (d)
helping to allocate strategy monies on an unsound basis.
None
of these
alternatives would appear desirable.
Next, the amount of tolerable uncertainty should be considered.
It
might be measured by assuming
analogous
a
to the
problem by
(not intuitively obvious) that
An alternative and
measure of the dispersion
uncertainty of 'almanac' studies.
itself is the interquartile range,
of responses.
it is
range
If this
is five
seven years, when the median
is
a
years when the median date
1975,
is at
1980 and ten years at 1988, there
may
be a point in time where the range places limitations on the usefulness of
the forecast.
In
other words, assessments of the utility of forecasts
with various interquartile ranges (e.g.
made prior
should be
or with other techniques.
it
quantitative.
a 1980
median forecast
f 5 years)
establishing a Delphi based planning system.
to
Another area of interest
modeling,
,
is the
Since Delphi
would be an advantage
question of utilizing Delphi alone
is a
technique based on subjective
to integrate it
with those that are
more
This would provide links with historical data and help to
yield a stronger forecast.
In general,
Delphi could be well integrated with
trend analysis, mqpping, and monitoring of the technological environment.
Two
additional
comments
of
use in the design of a Delphi forecast
are directed at the consideration of the research planning or industry
development cycle and the stratification of the forecast.
cycle
is
The research
important by supplying dates that will serve as targets upon which
to focus a forecast.
helpful dates, but
The industry development cycle can also provide
may
be particularly important by suggesting aspects of
technological developments that should be explored by the forecast,
capital funds constraints, sources and opportunities.
is the
breakdown
of all
e. g.
,
Forecast stratification
phases of the technology of interest into overlapping
subgroups that may, in turn, be addressed by groups of self-f«tted experts.
The overlapping
will
permit the development of internal consistency, while
the use of subgroupings will yield greater accuracy.
Aside from technical matters, one of the advantages of Delphi
that
it
stimulates thinking and involves
process.
management
This by itself could well be enough
In
summary,
in the forecasting
to justify its
use.
then, a consideration of the behavior of Delphi
combined with an awareness of
its
development should indicate the need
for a certain amount of caution in applying the technique too vigorously.
is
-^
Finally,
it
\
V
should also be apparent that Delphi
can be used
forethought.
to
is a
very useful
tool,
which
considerable advantage when applied with knowledge and
REFERENCES
(1)
Cetron, Marvin
J.
and Chri stine Ralph, Industrial Applications of
Technological Forecasting
(2)
Dalkey, M.
Method
John Wiley
Inc.
New
,
York,
1971.
and Olaf Helmer, "An Experimental Application of the Delphi
Use
to the
of
Experts", Management Science
,
TIMS, Baltimore, Md.
,
V.q ^M.3, Aprrl.
1963.
(3)
,
Kaplan, A.,A. L. Skogstad, M. A. Girshick, "The Prediction of Social
and Technological Events", Public Opinion Quarterly
,
Princeton, N.J.
,
Winter, 1950.
(4)
and Marvin Cetron, "SEER:
Bernstein, G. B.
A
Delphic Approach Applied to
Information Processing", Technological Forecasting
V.
(5)
1,
N.
Gordon, J.
Society,
(6)
1.
J.
,
"Forecasters Turn
Bethesda, Md.
,
V.
1,
N.
to Delphi",
1,
The Futurist
,
,
World Future
February, 1967.
"Corporation Issues Major Forecast of Future Technolgy", The Futurist
Pyke, Donald L.
Industrial
(8)
American Elsevier Pub.
June, 1969.
World Future Society, Bethesda, Md.
(7)
,
,
,
V.
1,
N.
1,
,
October, 1967.
"Mapping- A System Concept for Displaying Alternatives",
Management Center, Austin Texas,
1970.
"Delphi Studies as an Aid to Corporate Planning", Industrial Management
Center, Austin, Texas, 1970.
(9)
"Tire-Design Engineer Turns
ing
(10)
,
December
Dalkey, N.C.,
October, 1967.
21,
to
Technological Forecasting", Product Engineer
1970.
"Delphi",
RAND
Corp.
,
Santa Monica, Calif., P-3704,
(11)
Gordon, T.
and Robert H. Ament, "Forecasts
J.
Some Technological
of
and Scientific Developments and Their Societal Consequenses", The Institute
for the Future,
(12)
,
R-6, September, 1969.
Enzer, Selwyn, "A Case Study Using Forecasting as a Decisionmaking Aid",
The
(13^
Middletown, Conn.
Institute for the Future,
Gordon, T.
J.
"A Study
,
Institute for the Future,
Middletown, Conn.
of Potential
Changes
Middletown, Conn.
,
,
V.
V.
(15)
"
V.
The
(17)
(18)
-1,
April, 1969.
R-2
2,
"
R-3
3,
Raul de and Olaf Helmer, "Some Potential Societal Developments",
,
Institute for the Future,
Middletown, Conn.
Gordon, T' J. and Olaf Helmer
Appendix
R
1,
"
Bregatd,
1969.
Employee Benefits", The
in
(14)
(16)
WP-2, December,
to ref.
"McGraw
Hill
,
,
R-7, April, 1970.
"Report on a Long
range Forecasting Study"
-
26.
Survey
of
Technological Breakthroughs and Widespread
Applications of Significant Technical Developments", Department of
Economics,
(19)
McGraw
Hill Publications
Company.
/f
Parsons, P. and W. Williams, Forecast 1968-2000
Developments and Applications, Nyropsgade
47,
6
^•
of
1602
Computer
Copenhagen V, Denmark,
1968.
(20)
Helmer, Olaf and Nicholas Rescher, "On the Epistemology
Sciences",
Management Sciences
,
TIMS, Baltimore, Md.
,
of the Inexact
Vol.
6,
No.
1,
October 1959.
(21)
Lanford,
HW.
,
"A Synthesis
of
Technological Forecasting Methodologies",
manuscript, Wright State University, Wright-Patterson AFB^Ohio.
unpublished
unpul
WO
- ^^
-
(22)
Encyclopedia Brittanica, Vol.
(23)
Brown, Bernice and Olaf Helmer, "Improving the Reliability
(24)
From
a
Sept.
1964.
Consensus
"Delphi".
y^,
RAND
of Experts",
Corp.
Dalkey, N.
From
"Analyses
,
a
,
Vol.
No.
1,
Helmer, Olaf, Social Technology
,
Vol.
1,
No.
Basic Books Inc.
,
,
,
P-2986,
House
Iliffe
,
1969
Sept.
5,
Group Opinion Study", Futures
Science and Technology Publications Ltd.
(26)
Santa Monica, Calif.
,
Dalkey, N. "An Experimental Study of Group Opinion", Futures
Science and Technology Publications Ltd.
(25)
Estimates
of
House
Iliffe
,
Dec. 1969
S,
New
York, 1966.
James R. ed. Technological Forecasting for Industry and
Government particularly, "The Delphi Method - An Illustration (Bright),
(27^ Bright,
,
,
,
"Technology, the Chicken- Corporate Goals, The Egg (Harper Q. North).
(28)
Gordon, T.
J.
,
"Cross-Impact Matrices", Future
Science and Technology Publications Ltd.
(29)
Gordon, T.
Vol.
1,
House
Iliffe
iNo.
6.
,
Dec. 1969.
and H. Hayward, "Initial Experiments with the Cross
J.
Impact Matrix Method
of Forecasting",
Technology Publications Ltd., Vol.
(30)
,
,
Kovac, Frederick
J.
,
1,
Future
No.
2,
,
Iliffe
House Science and
Dec. 1968.
"Technology Forecasting-Tires", Chemical Technology
Jan. 1971.
(31)
Ament, Robert H.
1969",
Vol.
2,
Future
No.
1,
,
,
"Comparison
Iliffe
of Delphi
Forecasting Studies
in 1964
House Science and Technology Publications
Mar. 1970.
Ltd.
,
and
,
'22-
(3Z)
Dalkey, N. B.
Brown
S.
,
Cochra^ "Use
Improve
of Self-Ratings to
Group Estimates", Technological Forecasting. American Elsevier Pub. Co.
New York, Vol. 1, No. 3, Mar. 1970
,
(33)
Martino, Joseph P.
"The Precision of Delphi Estimates", Technological
,
Forecasting American Elsevier Pub. Co.
(34)
New
,
,
March
,
York,. Vol.
1
No.
,
3,
1970.
Martino, Joseph P.
Forecasting
,
,
"The Logncrmality
of Delphi Estinnates,
American Elsevier Pub. Co.
,
New
York, Vol.
Technological
",
1,
No.
4,
Spring 1970.
(35)
Blackman, A. Wade, "The Use of Bayesian Techniques
Teclmological Forecasting Vol. 2, No. 3/4^ HH.
in Delphi
Forecasts,
"
,
Additional
Kovac, Frederick
J.
Sources;
"Technology Forecasting- tires", Chemical Technology,
Jan. 1971, P. 18-23.
Fredriksson, Ann Littke-Porsson, "An Application
Landergren,
Ulf; Stig
of the Delphi
Technique
to a
Forecast
of the Field of
Ceramics- a
Commission for
Europe, Seminar on Technological Forecasting, Warsaw (Poland), 7-12
Methodological Experiment,"
United Nations Economic
Dec. 1970.
"Institute for the
The Futurist
,
August
1968.
Future Will Begin Operations in Connecticut
World Future Society, Bethesda, Md.
,
Vol.
2,
this Fall"
No.
4,
,
•7^
NOV*
Si
Il
'%tf^I)u,
DEC
K8
5
3
7g
•«'*
Lib-26-67
51'
DD3
TDfi
3
b
^'7(
542
'il7
3
DD3
1 Da
=!
E
-7/
53D
,,V'^'
3T06O 003 ^02
54fi
HD2B
Mil Ll6RA«itS
No
003 TOE 56T
3 IDfiD
MIT LIBRARIES
"
"
II'
-•7/
Si*?-
a75 S7E
TOflD 00 3
3
MIT LIBRARIES
03 fibT
TOflO
3
wn
3m
^^^'>/
aSBARIES
^17-7/
3
TD60 003
fi75
EMT
'^18-71
3
TOfiO
003
Mn
fibT
3E5
LIBRARIES
^/^'7'
TOaO 003 a75 223
3
521>'7/
3
Toa
3
a7S 531
MIT LIBRARIES
5ZI
3
TOaO 003
--7/
Ob 135
MIT LIBRARIES
?il-7'
3
Toa D 003 a75 lai
.T.
.
xr-