Policy Sciences 32: 225-245,1999.
1999 Kluwer Academic Publishers. Printed in the Netherlands.
225
A study of how individuals solve complex and
ill-structured problems
RONALD FERNANDES 1 & HERBERT A. SIMON 2
'//. John Heinz 111 School of Public Policy and Management, 'Department of Psychology, Carnegie
Mellon University, U.S.A.
E-mail: ronald(a andrew.cmu.edu
Abstract. A number of factors cause individuals to use diverse strategies to solve problems. This
paper presents a methodology for examining these differences in strategy. Verbal protocols are
elicited to collect data on the cognitive processes occurring during problem solving. These data,
codified into propositiona] representations, and non-parametric statistical comparisons are then
used to evaluate the significance of strategy differences. These strategies are then mapped with
dynamical graphs, with which we examine the task-independent and the task-specific cognitive
representations the participants used. As an illustrative example we apply this methodology to
study the influence of two contributing factors, professional training and national culture, on the
strategies adopted by professionals to solve a complex and ill-structured problem (hunger in a
country). The problem-solving strategies of professionals from different countries and trained in
architecture, engineering, law or medicine are analyzed to show some intriguing differences in the
general strategies adopted by individuals belonging to different professions, and the outcomes from
using these strategies.
1. Introduction
There has been a resurgence of interest in understanding how individuals solve
real-life policy problems. Such problems are usually complex and ill structured,
and research has been hindered in the past by inadequate empirical and analytical tools for isolating strategies used during the problem-solving process.
New developments in software now permit a finer-grained analysis of data
across a larger number of participants, including the use of non-parametric
statistical methods. We propose in this research note to describe briefly the
methodology and results of our research. A more detailed report on our findings is available from the authors on request.
Policy problems have most of the characteristics of complex and ill-structured problems. Funke (1991) has defined complex problems as having the
following features:
1. Intransparency: only knowledge about the symptoms is available, only
some variables lend themselves to direct observation, or, the large number of variables requires selection by the problem solver of a few relevant
ones.
2. Polytely: multiple goals may be present that could interfere with each other.
226
3. Situational complexity: there are complex connectivity patterns between
variables, and
4. Time-delayed effects: not every action shows immediate consequences.
Problems also vary along a continuum between the ill structured and the well
structured. The degree of definiteness depends upon the power of the problemsolving techniques available. Well-structured problems are those having:
1. A definite criterion for recognizing solutions and a mechanizable process
for applying that criterion.
2. At least one problem space in which the successive problem states may be
represented.
3. A structure wherein attainable state changes (legal moves) and considerable moves can be represented in a problem space as transitions.
4. A representation for any knowledge a problem solver can acquire, in one
or more problem spaces.
5. A reflection in the state changes, of the laws that govern the external
world and the effects upon a state of applying any operator, and
6. Basic processes that require only practicable amounts of computing and
only information that is available with practicable amounts of search
(Simon, 1973).
The differences in problem-solving strategies among individuals or groups can
be explained by a phenomenon called identification (Simon, 1997). A person
identifies with a group when, in making a decision, he evaluates the several
alternatives of choice in terms of their consequences for the specified group'.
This leads to decision-makers acquiring a representation of a problem that
focuses attention on operative goals, and interpreting them in terms of the
partial information attended to. When presented with a complex stimulus a
person perceives in it what he or she is ready to perceive; the more complex or
ambiguous the stimulus, the more the perception is determined by what is
already 'in' the individual and less by what is 'in' the stimulus. Thus identification can explain the adoption of available general problem-solving strategies by
individuals to solve complex problems, and the preference by individuals for
specific cognitive strategies over others.
Identification based on professional, ethnic or other characteristics can
cause individuals to apply problem-solving strategies that match the goals or
norms of the group identified with. These strategies can also be due to identifying a problem as relevant to a group's expertise. Professional identification
could contribute to the formation of general problem-solving strategies due to
the inculcation of "best practices' during the course of professional education. The limited focus of attention of professionals could contribute to these
strategies. These strategies could also occur due to a self-selection of individuals into professions they could identify with i.e. that match their cognitive
preferences.
227
Thus Altmeyer (1966) has shown that the cognitive abilities of undergraduate arts and science students change over the course of their professional
education. Engineering majors enhanced their analytical and logical reasoning
abilities, but showed a decline in some of their imaginative abilities; while arts
majors enhanced their imaginative abilities but exhibited a decline in some of
their analytical and logical reasoning abilities. Specialization of study leads to
differentiation of style, and mature, capable and motivated students display
very distinct thinking habits that are correlated with their choice of major field
(Doktor, 1969). Winter et al. (1981) document abundant evidence of cognitive
changes in individuals during the course of training in the liberal arts at the
undergraduate level.
Voss et al. (1983) investigated how experts and novices solved a problem of
low crop productivity in Soviet agriculture, an ill-structured social science
problem. One salient difference between experts and novices was in the time
taken to develop problem representations, with experts spending more time to
do so. In the solution process, experts also typically proposed relatively fewer
but more abstract solutions and spent considerable time in developing arguments related to the solutions. Novices on the other hand, proposed more
and simpler solutions with very little argument development. Voss and Post
(1988) then compared strategies used by these experts with those used by
magistrates and physicians as they solved problems in their field. They found
distinct similarities in the problem representational process across all cases.
There were marked similarities in the use of general problem-solving strategies
by experts, the most frequently employed method being decomposition. However, during the solution process, Soviet experts departed substantially from
magistrates and physicians. While for magistrates and physicians the solution
was presumed to be stated when the representation was established, Soviet
experts typically began the solution process by advancing a relatively abstract
solution, then a solution for each of the component sub-problems and finally
an integration of these solutions into a solution covering all aspects of the
problem.
Amsel et al. (1991) examined whether lawyers displayed a style of reasoning
distinguishable from the style that other professionals such as psychologists
exhibited. They showed that while lawyers relied primarily upon past-oriented
and diagnostic causal inference rules as represented by counterfactual rules,
psychologists preferred statistical evidence, the generation of hypotheses and
the evaluation of internal and external validity of an experimental design to
make causal inferences. Amsel et al. also suggested that the processes of
training in law and psychology induced professional differences in causal
reasoning.
Of the various methods that have been used to study individual problemsolving in complex and ill-structured situations, the approach used by Voss and
others (1983) was the most similar to ours in spirit. The data collected from
concurrent verbal protocols was segmented and classified into argument categories first proposed by Toulmin (datum, claim, warrant, backing, qualifier
228
and rebuttal). This approach gives us a rich insight into the problem-specific
strategies used during problem solving, but tends to obscure general strategies
used to represent or solve the problem. It also does not lend itself easily to
statistical evaluation of the differences. Our methodology, described in the next
section, can be applied to examining individual and/or group differences in
task-independent and task-specific strategies used for solving complex and illstructured problems.
2. Data collection, coding and verification
We begin by describing a complex and ill-structured situation. Each participant
is instructed to read the problem and then "think aloud' while solving it within
a limited time. The response is audiotaped without interaction between the
experimenter and the participant during the experiment. The audiotaped protocols are transcribed verbatim, segmented into cognitive chunks or actions
identifiable as separate units of thought, and encoded as one of nine basic
actions or nine meta-actions.
The nine basic actions were:
1. Recall: To recall facts specifically mentioned in the problem.
2. Read: To read words, phrases or statements from the problem.
3. Assume: To take for granted or suppose some fact or perceived fact not
mentioned in the problem and to use it.
4. Know: To be certain of some knowledge and to use it.
5. Infer: To conclude from evidence or premises or to adopt as a logical
consequence.
6. Evaluate: To fix the value of; or to examine and judge, information
specified in the problem.
7. Calculate: To calculate the numerical value of information.
8. Query: To utter a question, inquiry or a doubt, and
9. Recommend: To counsel or advise that something be done to solve the
problem.
Meta-actions describe or refer to any of the basic cognitive actions. They are
usually statements of the participant when he is thinking about the problemsolving process but not about specific problem details. Eighteen distinct actions
(9 actions and 9 meta-actions) are used to encode, at a prepositional level, the
thoughts expressed by each participant during the problem representation and
solving process. Independent coding can be carried out and the inter-coder
reliability verified.
229
3. Data analysis
The preliminary analysis of the protocols can be carried out in three stages:
1. Analysis of task-independent cognitive processes to determine general
problem-solving strategies.
2. Task specific analysis for insights into how individuals solve a specific
problem, and
3. Synthesis of the analyses to examine how general problem-solving strategies affect problem representation and solution outcomes.
3.1. Task independent analysis
Each protocol can be encoded using software to facilitate the process. Protocol
Analyst's Workbench (PAW; Fisher, 1991) is one such software tool that is
independent of the problem domain. The interactive feature of PAW facilitates
encoding that helps preserves the structure and content of the original transcript. PAW creates a matrix for each individual showing the frequency of use
of each action in the protocol, and a table of percentages of each action used
while solving the problem. Analysis of these percentages indicates the relative
importance of different actions. PAW also gives the frequency of transitional
and reflexive ties. Transitional ties occur when one action precedes another,
reflecting the relations between the cognitive actions used to solve the problem.
Thus if the action 'infer' is followed by the action 'recommend' it is recorded as
a transition from the one to the other action. Reflexive ties occur when one
action is followed by the same action, an example of which would be one 'infer*
statement followed by another 'infer' statement where both statements are
substantially different in content. Transitional and reflexive ties enable us to
assess an action's weight in a flow model of the problem representation and
solution process. They can also be used to set a quantitative threshold for
considering a tie to be part of a cognitive process, to eliminate noise.
The weights or percentages of these ties for each individual's protocol
can then be entered in matrix form in any matrix manipulation software such
as UCINET (Borgatti. Everett and Freeman, 1992) to create an individual
matrix showing the percentage of transitional and reflexive ties on actions for
each individual. Each matrix so obtained can be tested for similarity with the
matrices of other individuals, using the correlation between an individual's
matrix, and all the other individual matrices. This gives a data matrix showing
the correlation between each individual and every other individual. This data
matrix measures individual similarity or differences in transitional and reflexive
ties.
In order to test whether the results obtained are consistent with our hypothesized differences in characteristics between individuals, the data matrix can
then be compared for similarity with a characteristic structure matrix. This
230
matrix has cells with 1 's if the pair of individuals share a similar characteristic
and O's if they do not. In making statistical comparisons, structural autocorrelation may occur in the data matrix, due to the dyadic nature of relationships between individuals and non-independence of observations, leading to
biased results. However Quadratic Assignment Procedure or QAP 1' 2, a nonparametric method of statistical analysis available in UCINET has been shown
to give relatively unbiased results in such situations (Krackhardt, 1988).
Other characteristics might also coincidentally influence differences among
individuals in their problem solving strategies as represented by the data matrix. In such a situation QAP can be used to compare3 the similarity between
the data matrix and the new characteristic structure matrix. The magnitude of
the correlation coefficient between the two matrices and the level of significance would then indicate mutual influence. On the other hand the factors we
hypothesize as influencing problem solving strategies may be correlated. QAP
can then be used to determine the collinearitv between the two characteristic
structure matrices. If this collinearity is severe, we can separate out the relative
effects of both characteristics on the observed differences in problem-solving
strategies between individuals. QAP multiple regression analysis4 can be performed using the data matrix as the dependent matrix and the two characteristic structure matrices as independent matrices.
Having determined strategies used by individuals with specific characteristics or belonging to certain groups we can ask: Can these specific strategies
be identified? We found helpful a graphical method for visualizing some details
of the problem-solving strategies adopted by professionals. To make graphical
comparisons, KrackPlot (Krackhardt et al., 1994) can display transitional and
reflexive ties and the centrality and proportion of these ties for each action. A
flow diagram displays the sequence of actions each individual follows while
solving the problem. We found it appropriate5 to draw individual flow diagrams
that showed only those sequential pairs of successive actions with transitional
and reflexive ties that were common to individuals within a specified group.
Our next step is to understand how these general problem-solving strategies
could influence how individuals actually go about solving a complex and illstructured problem. The next two sub-sections indicate how general problemsolving strategies might influence the task specific outcomes and recommendations given by individuals.
3.2. Task specific process analysis
Task specific process analysis requires an understanding of the domain knowledge that professionals use in problem solving. It asks whether professionals
differ in their use of information to solve complex ill-structured problems.
Problem behavior graphs (Newell and Simon, 1972) display the sequence of
actions within a protocol and changes in representation of problem states in the
individual's mind. A careful examination of the content of these graphs reveals
231
similarities and differences in how individuals belonging to similar and different groups solve problems.
3.3. Synthesis of results
The most critical section of the analysis is the synthesis of what we know about
the general problem-solving strategies used by individuals, and the specific
outcomes of the problem-solving process. If done appropriately this synthesis
can determine whether general problem-solving strategies play any part in
determining problem representation and solution process and the nature of the
solutions offered. A successful synthesis relates what we know about individual
or group characteristics to specific strategies used by that individual or group.
We will demonstrate the use of this method in the next section, using an
example of the analysis of strategies in a complex and ill-structured problem.
4. Illustrative example: Policy problems as complex and ill-structured
problems
The problem of hunger in a country was chosen as an example. The problem of
hunger meets the criteria of complexity and ill structuredness in many ways,
so that individuals could use both domain-specific knowledge and general
problem-solving strategies to solve the problem. There could be a wide variation in the knowledge each individual had or applied to the problem, derived
both from information provided in the problem statement, and from external
knowledge.
4.1. Policy problem
The policy problem given to the participants consisted of the role to be played
by the problem-solver followed by a brief description of the problem:
As a senior policy maker in your country you are called upon to advise an
international organization about what is needed to solve a specific policy
problem in a country called Hungeria. Please explain what other information
you might need to solve this problem and then what your recommendations
would be?
Brief description of the problem:
Hungeria, with a population of 26 million, has nearly four million people who
exist below the poverty line and around one million people who are undernourished and hungry because they do not have sufficient food to eat. In
232
estimating the numbers of people going hungry the Food and Agriculture
Organization of the United Nations (FAO) uses as its criterion the energy
intake level at which a person can barely survive which is a daily calorie intake
below 1.2 basal metabolic rate (around 2100 Calories). Hungeria has a Gross
Domestic Product (GDP) of $638 billion and is a net food exporting country.
The birth rate in Hungeria is 15 per thousand and the death rate is 8 per
thousand.
4.2. Specifying the complexity and ill-structuredness of the problem
The problem of hunger frequently arises from complex linkages between the
production, distribution and consumption of food within the context of political, social and economic institutions (Timmer, 1983). Lack of income and longterm employment can lead to hunger even in relatively developed countries. The
food policy of a country is also constrained by the international economy,
through its impact on balance of payments, foreign exchange rates and the
import and export of food commodities. Instability in international food prices
can seriously affect domestic food policies.
Some of the actions that may help address the problem of hunger are:
1. Redistribution of assets: Highly unequal ownership of assets frequently
affects food production and consumption. Land reform may achieve a
redistribution of assets, as may taxation of the rich and subsidies to the
poor.
2. Growth in income for the poor: Growth strategies may provide an increase
in real incomes for the poor by providing new employment opportunities.
3. Reduced discrimination against certain social or demographic groups: Job
opportunities may be improved for specific social or demographic groups
by reducing discrimination against them.
4. Increase in the production offood: An increase in the production of food
by a country may be achieved through technical change.
The problem of hunger in a country therefore meets the criteria of complexity
specified earlier. In order to ensure that it was also ill structured, the problem
was designed with some 'contradictions'. This would induce multiple problem
representations both in terms of problem structure and domain knowledge.
The first "contradiction" was in the facts that Hungeria had a Gross Domestic Product (GDP) of S638 billion and a population of 26 million. This works
out to a per capita GDP of around 524,000, among the highest in the world.
Another 'contradiction' was the statement that Hungeria was a net food exporting country, implying that food production was not a serious problem. A third
'contradiction' was in the rather moderate birth rate and death rate, which were
representative of a highly developed country. Fourth, no information was given
about the occurrence of any natural disaster that could have caused the prob-
233
lem. These 'contradictions' eliminated various potential caus
es of hunger in the
country. A good solution would require that professionals
either question that
information or make suitable assumptions to assure cons
istency before offering
recommendations. The information provided in the prob
lem was sufficient to
support recommendations consistent with the 'contradictio
ns', along two directions:
1. Targeted food subsidies could provide immediate relie
f to the hungry (e.g.
food stamp programs, monetary support programs and
food-for-work
programs). Given the high per capita GDP, this was
an appropriate
solution of the problem in the short run.
2. Long-term recommendations might include redistrib
ution of income
through tax policies or through productive jobs for peop
le below the
poverty line.
4.3. Data collection, coding and verification
The professions chosen were architecture, medicine, law
and engineering. Each
of the participants6 held undergraduate degrees in these
four fields. Each participant J was given 30-45 minutes to solve the problem
while thinking aloud.
The coding was carried out according to the methodol
ogy described earlier
(9 basic and 9 meta-actions). To test reliability of this
coding, independent
coding was carried out on a random sample (20%) of
the encoded actions.
Intercoder reliability for this sample was 96%. Finally Prot
ocol Analysts Workbench (PAW) was used to encode each protocol 7 systemati
cally.
4.4. Task independent analysis
PAW was used to generate the percentages of actions used
by each professional.
The results are shown in Table 1. The percentages in
Table 1 indicate that
professionals used substantially more basic actions than meta
-actions. However,
architects were an exception among the professional grou
ps in their rather
extensive use of meta-actions. PAW was then used to
analyze the relative
proportions of transitional and reflexive ties. With a total
of 18 types of actions
(9 basic, 9 meta) for each professional we obtained a 18 x
18 square matrix like
that in Table 2.
Each matrix was then tested for similarity with the matrice
s of all the other
professionals using UCINET. This produced an 8 x 8 data
matrix of the correlation coefficients as displayed in Table 3. Table 3 show
s that individuals
belonging to the same profession have higher correlation
coefficients in general
than individuals belonging to dissimilar professions,
with two exceptions.
Engineer 2 (E2) shows a higher correlation with Lawyer
1 (LI) and Lawyer 2
(L2) than he does with Engineer 1 while Physician 2
(P2) shows a higher
234
Table I. Relative importance of actions used by professionals (in %).
Actions
Assume
Calculate
Evaluate
Infer
Know
Query
Read
Recall
Recommend
M-Assume
M-Calculate
M-Evaluate
M-Infer
M-Know
M-Query
M-Read
M-Recall
M-Recommend
Participants
Architect 1
Architect 2
Physician 1
Physician 2
Engineer 1
Engineer 2
Lawyer
1
Lawyer
2.9
1.5
13.2
10.3
16.2
19.1
6.6
3.9
14.5
2.6
6.6
17.1
1.3
2.6
13.2
3.9
5.3
5.3
11.8
0.0
3.9
1.3
0.0
0.0
3.3
3.3
0.0
13.3
6.7
13.3
21.7
18.3
8.3
0.0
0.0
1.7
5.0
0.0
3.3
0.0
0.0
1.7
3.2
1.6
11.3
4.8
8.1
25.8
12.9
9.7
4.8
0.0
0.0
0.0
12.9
0.0
1.6
0.0
0.0
3.2
12.2
0.0
9.8
17.1
4.9
22.0
0.0
2.4
19.5
0.0
0.0
4.9
2.4
0.0
2.4
0.0
0.0
2.4
7.8
3.1
4.7
20.3
14.1
6.3
10.9
0.0
23.4
0.0
4.7
3.1
1.6
0.0
0.0
0.0
0.0
0.0
2.5
1.9
15.4
16.7
19.8
0.6
3.1
6.8
20.4
0.0
0.0
3.7
3.7
0.0
1.2
0.0
0.6
3.7
14.9
0.0
2.1
6.4
36.2
5.3
0.0
1-1
28.7
0.0
l.l
3.2
I.I
0.0
0.0
0.0
0.0
0.0
4.4
5.9
5.9
0.0
4.4
4.4
7.4
0.0
0.0
0.0
0.0
4.4
Total meta-actions
20.6
31.6
11.7
17.7
12.2
9.4
13.0
5.3
Total basic actions
79.4
68.4
88.3
82.3
87.8
90.6
87.0
94.7
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Total
correlation with Architect I (Al) and Architect 2 (A2) than with Physician 1
(PI). In order to test whether the results were consistent with differences in
professional education among the participants, the data matrix was compared
for similarity with a profession structure matrix (Table 4), which had entries of
1 if a pair of participants belonged to the same profession and 0 if they belonged
to different professions.
QAP was used to calculate the correlation coefficient between the data
matrix and the profession structure matrix. Table 5 shows that the correlation
was 0.37. The probability of a correlation as large or larger than 0.37 with a
random matrix was 1.1% indicating that the correlation was significant at the
0.05 level.
As our sample included professionals who had obtained their professional
education outside the U.S., these differences between professions might be due
to other cultural differences. In order to examine the significance of such differences, a country structure matrix was created (Table 6) that had 1 's if the pair of
participants were educated in the same country and O's if they were educated
in different countries. QAP was used to compare the similarity between the
data matrix and the country structure matrix. Table 7 shows that the observed
correlation between the two matrices was 0.26. The percentage of random
Recom
Recal
Read
Query
M-Recal
M-Recom
M-Que
M-Rea
M-F.va
M-Inf
M-Kno
M-Cal
AsMim
('alcu
F.valu
Infer
Know
M-Ass
Calc
3
2
1
Eval
2
1
1
Infer
2
2
4
Know
2
M-Eva
|
|
----2
- ..
1
M-Inf
•
M-Reca
M-Reco Query
-
|
1
1
I
9
7
II
1
0
.1
.1
4
21111
2
0
0
0
0
3
Recal
14
3
4
4
68
24
4
13
3
I
0
13
o
0
3
5
o
3
I
I
I
|
12
I
9
7
II
0
Recom
I
6
_._...-_..._...
__
........_....
..(.......
Read
212
--.--
M-Rea
.........
M-Know M-Qne
12
.......
|
2
I
..._..........
III
I
M-Cal
..
.....
M-Ass
..._...-
„
Assu
7'dhfi' 2. Transitional and reflexive ties matrix for Architect I.
236
Table 3. Data matrix.
1
2
3
4
5
6
7
8
Al
A2
PI
P2
El
E2
LI
L2
1
Al
2
A2
3
PI
4
P2
5
El
6
E2
7
LI
8
L2
1.00
0.55
1.00
0.15
0.0?
1.00
0.44
O.J8
0.30
1.00
0.43
0.45
0.14
0.30
1.00
0.38
0.30
0.25
0.13
0.42
1.00
0.38
0.23
0.17
0.14
0.36
0.68
1.00
0.36
0.27
0.19
0.09
0.24
0.63
0.62
1.00
Bold - significant' at 0.01 level. Bold italics - significant at 0.05 level. Italics - significant at 0.10 level.
Table 4. Profession structure matrix.
1
2
3
4
5
6
7
8
Al
A2
PI
P2
El
E2
LI
L2
1
Al
2
A2
3
PI
4
P2
5
El
6
E2
7
LI
8
L2
1
1
I
1
0
0
1
1
0
0
1
1
0
0
0
0
1
1
0
0
0
0
1
I
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
8
L2
Table 5. QAP correlation between data matrix and profession structure matrix.
Observed value
Average
Standard deviation
Proportion as large
Proportion as small
Correlation
Matches
0.372
0.003
0.178
0.011
1.000
0.000
0.000
0.000
1.000
1.000
Table 6. Country structure matrix.
1
Al
1
2
3
4
5
6
7
8
Al
A2
PI
P2
El
E2
LI
L2
2
A2
1
0
0
1
0
0
0
0
0
1
0
0
0-0
0
0
3
PI
4
P2
5
El
6
E2
7
LI
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
1
1
0
0
0
M)
0
0
1
1
237
Table 7. QAP correlation between data matrix and country- structure matrix.
Observed value
Average
Standard deviation
Proportion as large
Proportion as small
Correlation
Matches
0.258
0.002
0.175
0.102
0.899
0.000
0.000
0.000
1.000
1.000
Table 8. QAP correlation between profession structure matrix and country structure matrix.
Observed value
Average
Standard deviation
Proportion as large
Proportion as small
Correlation
Matches
0.519
0.001
0.210
0.070
0.995
0.893
0.781
0.045
0.070
0.995
Table 9. QAP regression of data matrix on profession structure matrix and country structure matrix.
Independent
Intercept
Profession
Country
Regression coefficients
Un-stdized
coefficient
Proportion
as large
Proportion
as small
0.28
0.14
0.05
0.994
0.082
0.293
0.006
0.918
0.707
/^-square: 0.144. Probability: 0.077.
correlation's as large or larger than 0.26 was 10.2% indicating a correlation
between the data and the country structure matrix lower than that between
the data and profession structure matrix, and not statistically significant at the
0.10 level.
The profession and country structure matrices had a fairly high level of
collinearity. The QAP correlation coefficient between the two matrices was
0.52, which was significant at the 0. 10 level as shown in Table 8. In order to
separate out the relative significance of professional education and country
effects on the general problem-solving strategies used by professionals, QAP
multiple regression analysis was performed using the data matrix as the
dependent matrix and the profession structure and country structure matrices
as independent matrices. Table 9 shows the results of the QAP regression.
Profession appeared significant at the 0.10 level while country is not significant.
Therefore there is evidence of profession-specific general problem-solving
238
strategies, reflected in the transitional and reflexive ties between actions, that
are significantly similar for professionals belonging to the same profession and
differ across individuals belonging to different professions.
We then used the graphical method recommended earlier to view and com­
pare these general problem solving strategies. Shown below are the flow dia­
grams created using Krackplot (Figures 1 and 2) for two professional groups:
engineers and lawyers in our sample, along with our interpretation.
The common actions for engineers (shown in Figure 1) consisted of 5 out of
the 8 actions and no meta-actions. These 5 actions account for 81% of all
actions for Engineer 1 and 66% for Engineer 2. The actions Recommend, Infer;
Assume and Query also show reflexive ties indicating the repeated use of these
actions in solving the problem. No action appears to be truly central to the
process for either engineer.
The common actions for lawyers (shown in Figure 2) consisted of 6 out of
the 8 actions and no meta-actions. These 6 actions account for 77% of all
actions for Lawyer 1 and 94% for Lawyer 2. The actions Recommend and
Assume also show reflexive ties indicating the repeated use of these actions in
solving the problem. On the whole the action Know appears to be central to the
process with the most cyclic ties to other actions.
(
Interpretation
The lawyers and engineers tend to use a large proportion of Recommend
actions as compared to other professionals. However the process through
which they arrive at a Recommend action differs substantially. Lawyers show
a transition from actions Know and Infer to Recommend, while engineers
showed transitions from the actions Assume, Query and Evaluate to Recom­
mend. Another difference is that transitions between Assume and Infer, appear
for lawyers but not for engineers, although engineers use both these actions
more than lawyers do.
4.5. Task specific process analysis
Task specific process analysis relies upon the domain knowledge that profes­
sionals use. Problem behavior graphs were built to display the sequence of
actions and changes in representation of problem states in the protocols for
both physicians (extracts shown below), followed by our interpretation of the
graphs.
239
Engineer 1
H-QUERY
H-INFER
5
'
M-CMJCULMlf.
\
M-EIM1.1MTE
in
Engineer 2
M-QUERY
H-1HFER
H-CALJCUUATE
KNOW
H-EUM.IMTE
240
Lawyer 1
H-qUERV
H-INFER
«-CflLCUU»TE
M-EUM-IMTE
EUM-UATE
QUERY
Lawyer 2
H-QUERY
H-CM.CULATE
H-IMFER
17
ft-EUMLlMTE
EUM.UATE
QUERV
Fig. 2.
241
Extracts from problem behavior graphs: Physicians
Physician 1
Physician 2
46. Inferi people are going hungry and people
are undernourished
47. Inter: they are not carrying 2100 calories
SO. »«i leanenil; food should be given no the
people rather than exported
8. Infer: one million people fall in this
category
9. Infer: the cut-off level is 2100 calories
25. Calculate: per capita income-24.DOC dollars
per year
52. Recoeaendi foods costs should be low
54. XeeoBeead: production in the country
56. Kecaeviend: population control
60. » !. i emend; increase infrastructure of
country
26. Know: that is the per capita income for the
U.S.A
27. S-rtluatei how car. that be
28. Bvmloate: Not possible
29. Bymluee: figures are grossly inadequate to
offer any recommendations
54. Bmloatei birth rate:seems to be a pretty
stable population
55. Evaluate: 15 per thousand birth and 8 per
thousand death rate is fine
56. Infer: not a rapidly growing population
61. Met«-»»i leeMiiil: not enough data tc sake
serious recommendations
Population control, which has medical implications, was considered by both
physicians during the problem-solving process, with Physician 1 recommend
ing
population control as a solution. Physician 2 however critically considered
the
information given in the problem about the birth and death rates in the count
ry
and correctly inferred that population control was not the problem. They were
also similar in their attempt to focus on diagnosis of the problem with
an
inability to diagnose due to the 'contradictions' (especially Physician 2, state­
ments 29 and 61).
4.6. Synthesis of results
Do general problem solving strategies adopted by professionals play any part
in
determining the problem representation and solving process and the nature
of
solutions offered? Given below are some examples of use of these strategies
by
two professional groups: architects and lawyers.
Architects
The professional education architects receive trains them in architectural
de­
sign (Akin, 1986: pp. 176-177). The design process shares with policy problems
a complex and ill structured nature. Both architects sought a solution using
strategies of architectural design. One of the dominant strategies was the Query
Evaluate strategy shown below.
242
Query <==> Evaluate strategy
I
,
i
j
Architect 1
Architect 2
13. Kr*laat«i unfair distribution of income
14. Query: income distribution graph?
15. Query: land use details of country?
16. Query: potential for country to grow more
food?
17. Query: food eaten by the people?
34. >Md: the country is net food exporting
35. Infer: no real shortage of food in country.
36. Infer: food goes into the wrong channels
43. Queer: information on the administration,
bureaucracy and politicians in the country?
44. Query: political structure in the country?
45. XnlumCet futile to draft policy that would
not be implemented by bureaucrats
54. Infer: not a nourishment problem
23. Query: am I supposed to feed one million
people?
55. evaluate: the problem is probably deeper than
that.
58. X-Heoaemend: solution would have to be
discussed with other people.
24. Xvaluate: the best vay -o feed 1 million
people
29. XeceaBtnd: feed the undernourished
48. Xraluate: hard to say what types of job
creation
49. Query: skills of population?
50. Query: education levels of population?
51. Query: climate of country?
52. Query: geographic location?
53. Query: food consumption habits?
61. JU « : retraining issue-people are not
properly trained for available jobs
ml retraining
62. 1Ui i
63. »«iu»ei people are not properly equipped
for current job opportunities
basic training.
64.Recommend:
Thus Architect 1 used this strategy to determine that the problem was a wealth
distribution problem for which he needed more information. He queried land
use facts and food production facts that were not apt in the context of this
problem, as the country had net food exports. He however heeded this point
later on, which led him to realize the complexity of the problem (statement 55)
and to offer a meta-recommendation (statement 58) rather than a recommen­
dation. Similarly Architect 2 used this strategy to make a short-term recom­
mendation of feeding the undernourished (statement 29). However his long
term recommendations for creating jobs were restricted by insufficient infor­
mation (statements 49 to 53), and his consequent recommendations for creat­
ing jobs were based upon assumptions (statements 61 and 63) he made about
the skills of the people.
Lawyers
Lawyers are trained to choose between alternative views, while focussing on
facts that support their view (Wrightsman, 1987). The nature of their training
predisposes them to focus on conflict resolution rather than on discovering the
truth. Some elements of this professional training are reflected in the use of
the actions Know and Recommend in their problem-solving strategies. Their
primary problem-solving strategy used was the Know <=> Recommend strategy
with reflexive ties on the actions Know and Recommend as shown below:
243
Know <=> Recommend strategy
Lawyer 1
63 . tacoBaradi has to be a program for
producers
64. Know: World Bank won't support anything
against international or open markets
65. Knew: World Bank not a supporter of
subsidies
66. »«i iiiaiaml: a prograir. for auto-consumption
67. RaoooMad: program tc increase productivity
of land
Lawyer 2
21. Know: wealth accumulation in small
coranunities leads to some pressure for
redistribution
28. B»i miMiiil provide people with education of
practical skills
29. Know: this can be cheap though highly
effective
41. Knowi without businesses, principles alone
cannot sustain people
42. Jn n»«in1 use joint ventures to add value
to agricultural products
The strong Know <=> Recommend transitional tie for both lawyers appears to
assist them to make recommendations. Thus Lawyer I uses his knowledge of
the funding norms of the World Bank (statement 64) to foresee a problem for a
short-term program of auto-consumption (statement 66). Similarly Lawyer 2
uses his knowledge of how hungry people behave (statements 24 through 27)
to recommend education in practical skills as a way of solving the problem
(statement 28).
5. Conclusions
Our examination of the protocols suggests that the use of general strategies
appears to be sometimes helpful and sometimes harmful in enabling profes­
sionals to solve complex ill-structured problems. For example the use of the
Know <=> Recommend strategy by the two lawyers in our sample hindered
their ability to use information provided in the form of 'contradictions'. The
essential issue here seems to be determining when some general cognitive
strategies facilitate the problem solving process and when they hinder it.
A question that has interested researchers for some time has been: Are there
singular differences in the general problem-solving strategies of professionals
solving complex and ill-structured problems and can these differences be iden­
tified and measured? The methodology and analysis of this research note
appears to identify such differences and may be applied to other real-world
public policy problems. Using verbal protocol analysis, the sophisticated
software currently available to analyze large and complex protocols, and the
statistical and graphical tools described in this research note, valuable insight
could be gained into the decisions taken by professionals.
244
Acknowledgements
The authors are grateful for the comments of Linda Babcock, Danny Fernandes,
David Krackhardt, Denise Rousseau and the comments of two anonymous
referees on earlier drafts of this paper. Stephanie Mathews and Zhaoli Rong
provided invaluable research assistance. The usual disclaimers apply.
Notes
1. The algorithm testing the similarity of matrices proceeds in two steps. In the
first step it
computes Pearson's correlation coefficient (as well as the simple Matching coefficient)
between
corresponding cells of the two matrices. In the second step, it randomly permutes
rows and
columns of one matrix and recomputes the correlation. The second step is carried out
hundreds
of times in order to compute the proportion of times that the correlation based on
random
permutations is equal to or larger than the observed correlation calculated in step
1. If the
correlation is positive, a low percentage (<0.05) suggests that the observed similari
ty (as
indicated by the observed correlation) between the matrices is unlikely to have occurre
d by
chance (cf. UCINET IV Version 1.0 - Network Analysis Software: User's Guide by
Borgatti,
Everett and Freeman, 1992: pp. 127-128).
2. The comparisons ignore the diagonals of both matrices. This avoids an artificial
similarity due
to both matrices having all 1's in their diagonals.
3. See note 2.
4. QAP regression procedure performs a standard multiple regression across corresp
onding cells
of the dependent and independent matrices in the first step. In the second step, it
randomly
permutes rows and columns (together) of the dependent matrix repeatedly and recomp
utes the
regression, as described in Note 1.
5. Two factors motivated this decision. First, that this would act as an approximation
to eliminat­
ing any noise or false sequences between or within actions, from the diagram. Second
, we would
have established that individuals show higher levels of similarity within a group.
6. All the participants were male. There was some variation in the nature of the
professional
education each had received subsequent to completing his undergraduate degree
and their
current occupation, although the basic categories were restricted to professionals
who had
undergraduate and graduate degrees in those specific fields. The exceptions were one
lawyer
who was presently in a graduate program in public policy (Lawyer 1) and one
engineer,
currently in a graduate program in industrial administration (Engineer 2). Both
physicians
were practicing at a local hospital while the remaining participants were in graduat
e school
related to their field of undergraduate study. The professionals differed in the country
where they
did their undergraduate studies. Both physicians (India), both lawyers (Mexico), one
engineer
(France) and one architect (Switzerland) had non-U.S. undergraduate degrees.
7. Encoded protocols available on request.
References
Akin, O. (1986). Psychology of Architectural Design. London: Pion Limited, pp. 176-177
.
Altemeyer, B. (1966). 'Education in the arts and siences: Divergent paths,1 doctora
l dissertation.
Carnegie Mellon University.
Amsel, E., R. Langer and L. Loutzenhiser (1991). 'Do lawyers reason differently from
psycholo­
gists? A comparative design for studying expertise.' in R. Sternberg and P. Frensch, eds..
Complex
Problem Solving-Principles and Mechanisms, Hillsdale, New Jersey: Lawrence Erlbaum
Asso­
ciates, pp. 223-250.
245
Borgatti, E. and Freeman (1992). 'UCINET IV Version 1.0 - Network Analysis Software: User's
guide'.
Doktor, R. (1969). The development and mapping of certain cognitive styles of problem solving.'
doctoral dissertation, Stanford University.
Ericsson, A. K. and H. A. Simon (1983). Protocol Analysis: Verbal Reports as Data. Cambridge.
MA: The MIT Press.
Fisher, C. (1991). 'Protocol analyst's workbench: Design and evaluation of computer-aided protocol
analysis.' doctoral dissertation. Carnegie Mellon University.
Funke. J. (1991). 'Solving complex problems: Exploration and control of complex systems,' in R.
Sternberg and P. Frensch, eds.. Complex Problem Solving - Principles and Mechanisms, Hillsdale. New Jersey: Lawrence Erlbaum Associates, pp. 185-222.
Krackhardt. D.. J. Blythe and K. McGrath (1994). 'KrackPlot 3.0: An improved network drawing
program.' Connections 17 (2): 53-55.
Krackhardt. D. (1988). 'Predicting with networks: Nonparametric multiple regression analysis of
dyadic data,' Social Networks 10, pp. 359-381.
Newell, A. and H. A. Simon (1972). Human Problem Solving. Englewood Cliffs, N.J.: Prentice-Hall.
Schwartz, S. and T. Griffen (1986). Medical Thinking - The Psychology of Medical Judgment and
Decision-Making. New York: Springer Verlag.
Simon. Herbert A. (1997). Administrative Behavior, 4th edition. New York: The Free Press.
Simon. H.A (1973). The structure of ill-structured problems,' Artificial Intelligence 4, pp.181-201.
Timmer, P., Walter F. and Scott Pearson (1983). Food Policy Analysis. Baltimore: The Johns
Hopkins University Press.
Voss, J., S.W. Tyler and L. A. Yengo (1983). 'Individual differences in the solving of social science
problems,' in R. F. Dillon. R. R. Schmeck, eds., Individual Differences in Cognition, New York:
Academic Press.
Voss, J. F. and T.A. Post (1988). 'On the solving of ill-structured problems,' in M.T.H. Chi, R.
Glaser, and M. J. Farr, eds., The Nature of Expertis, Hillsdale, N.J.: Lawrence Erlbaurn Asso­
ciates, pp. 261-285
Winter, D.G., D.C. McClelland and A. J. Stewart (1981). A New Case for the Liberal Arts. San
Francisco, California: Jossey Bass Publishers.
Wrightsman. L. S. (1987). Psychology and the Legal System, Monterey, California: Brooks/Cole
Publishing Company.