Policy Sciences 32: 225-245,1999. 1999 Kluwer Academic Publishers. Printed in the Netherlands. 225 A study of how individuals solve complex and ill-structured problems RONALD FERNANDES 1 & HERBERT A. SIMON 2 '//. John Heinz 111 School of Public Policy and Management, 'Department of Psychology, Carnegie Mellon University, U.S.A. E-mail: ronald(a andrew.cmu.edu Abstract. A number of factors cause individuals to use diverse strategies to solve problems. This paper presents a methodology for examining these differences in strategy. Verbal protocols are elicited to collect data on the cognitive processes occurring during problem solving. These data, codified into propositiona] representations, and non-parametric statistical comparisons are then used to evaluate the significance of strategy differences. These strategies are then mapped with dynamical graphs, with which we examine the task-independent and the task-specific cognitive representations the participants used. As an illustrative example we apply this methodology to study the influence of two contributing factors, professional training and national culture, on the strategies adopted by professionals to solve a complex and ill-structured problem (hunger in a country). The problem-solving strategies of professionals from different countries and trained in architecture, engineering, law or medicine are analyzed to show some intriguing differences in the general strategies adopted by individuals belonging to different professions, and the outcomes from using these strategies. 1. Introduction There has been a resurgence of interest in understanding how individuals solve real-life policy problems. Such problems are usually complex and ill structured, and research has been hindered in the past by inadequate empirical and analytical tools for isolating strategies used during the problem-solving process. New developments in software now permit a finer-grained analysis of data across a larger number of participants, including the use of non-parametric statistical methods. We propose in this research note to describe briefly the methodology and results of our research. A more detailed report on our findings is available from the authors on request. Policy problems have most of the characteristics of complex and ill-structured problems. Funke (1991) has defined complex problems as having the following features: 1. Intransparency: only knowledge about the symptoms is available, only some variables lend themselves to direct observation, or, the large number of variables requires selection by the problem solver of a few relevant ones. 2. Polytely: multiple goals may be present that could interfere with each other. 226 3. Situational complexity: there are complex connectivity patterns between variables, and 4. Time-delayed effects: not every action shows immediate consequences. Problems also vary along a continuum between the ill structured and the well structured. The degree of definiteness depends upon the power of the problemsolving techniques available. Well-structured problems are those having: 1. A definite criterion for recognizing solutions and a mechanizable process for applying that criterion. 2. At least one problem space in which the successive problem states may be represented. 3. A structure wherein attainable state changes (legal moves) and considerable moves can be represented in a problem space as transitions. 4. A representation for any knowledge a problem solver can acquire, in one or more problem spaces. 5. A reflection in the state changes, of the laws that govern the external world and the effects upon a state of applying any operator, and 6. Basic processes that require only practicable amounts of computing and only information that is available with practicable amounts of search (Simon, 1973). The differences in problem-solving strategies among individuals or groups can be explained by a phenomenon called identification (Simon, 1997). A person identifies with a group when, in making a decision, he evaluates the several alternatives of choice in terms of their consequences for the specified group'. This leads to decision-makers acquiring a representation of a problem that focuses attention on operative goals, and interpreting them in terms of the partial information attended to. When presented with a complex stimulus a person perceives in it what he or she is ready to perceive; the more complex or ambiguous the stimulus, the more the perception is determined by what is already 'in' the individual and less by what is 'in' the stimulus. Thus identification can explain the adoption of available general problem-solving strategies by individuals to solve complex problems, and the preference by individuals for specific cognitive strategies over others. Identification based on professional, ethnic or other characteristics can cause individuals to apply problem-solving strategies that match the goals or norms of the group identified with. These strategies can also be due to identifying a problem as relevant to a group's expertise. Professional identification could contribute to the formation of general problem-solving strategies due to the inculcation of "best practices' during the course of professional education. The limited focus of attention of professionals could contribute to these strategies. These strategies could also occur due to a self-selection of individuals into professions they could identify with i.e. that match their cognitive preferences. 227 Thus Altmeyer (1966) has shown that the cognitive abilities of undergraduate arts and science students change over the course of their professional education. Engineering majors enhanced their analytical and logical reasoning abilities, but showed a decline in some of their imaginative abilities; while arts majors enhanced their imaginative abilities but exhibited a decline in some of their analytical and logical reasoning abilities. Specialization of study leads to differentiation of style, and mature, capable and motivated students display very distinct thinking habits that are correlated with their choice of major field (Doktor, 1969). Winter et al. (1981) document abundant evidence of cognitive changes in individuals during the course of training in the liberal arts at the undergraduate level. Voss et al. (1983) investigated how experts and novices solved a problem of low crop productivity in Soviet agriculture, an ill-structured social science problem. One salient difference between experts and novices was in the time taken to develop problem representations, with experts spending more time to do so. In the solution process, experts also typically proposed relatively fewer but more abstract solutions and spent considerable time in developing arguments related to the solutions. Novices on the other hand, proposed more and simpler solutions with very little argument development. Voss and Post (1988) then compared strategies used by these experts with those used by magistrates and physicians as they solved problems in their field. They found distinct similarities in the problem representational process across all cases. There were marked similarities in the use of general problem-solving strategies by experts, the most frequently employed method being decomposition. However, during the solution process, Soviet experts departed substantially from magistrates and physicians. While for magistrates and physicians the solution was presumed to be stated when the representation was established, Soviet experts typically began the solution process by advancing a relatively abstract solution, then a solution for each of the component sub-problems and finally an integration of these solutions into a solution covering all aspects of the problem. Amsel et al. (1991) examined whether lawyers displayed a style of reasoning distinguishable from the style that other professionals such as psychologists exhibited. They showed that while lawyers relied primarily upon past-oriented and diagnostic causal inference rules as represented by counterfactual rules, psychologists preferred statistical evidence, the generation of hypotheses and the evaluation of internal and external validity of an experimental design to make causal inferences. Amsel et al. also suggested that the processes of training in law and psychology induced professional differences in causal reasoning. Of the various methods that have been used to study individual problemsolving in complex and ill-structured situations, the approach used by Voss and others (1983) was the most similar to ours in spirit. The data collected from concurrent verbal protocols was segmented and classified into argument categories first proposed by Toulmin (datum, claim, warrant, backing, qualifier 228 and rebuttal). This approach gives us a rich insight into the problem-specific strategies used during problem solving, but tends to obscure general strategies used to represent or solve the problem. It also does not lend itself easily to statistical evaluation of the differences. Our methodology, described in the next section, can be applied to examining individual and/or group differences in task-independent and task-specific strategies used for solving complex and illstructured problems. 2. Data collection, coding and verification We begin by describing a complex and ill-structured situation. Each participant is instructed to read the problem and then "think aloud' while solving it within a limited time. The response is audiotaped without interaction between the experimenter and the participant during the experiment. The audiotaped protocols are transcribed verbatim, segmented into cognitive chunks or actions identifiable as separate units of thought, and encoded as one of nine basic actions or nine meta-actions. The nine basic actions were: 1. Recall: To recall facts specifically mentioned in the problem. 2. Read: To read words, phrases or statements from the problem. 3. Assume: To take for granted or suppose some fact or perceived fact not mentioned in the problem and to use it. 4. Know: To be certain of some knowledge and to use it. 5. Infer: To conclude from evidence or premises or to adopt as a logical consequence. 6. Evaluate: To fix the value of; or to examine and judge, information specified in the problem. 7. Calculate: To calculate the numerical value of information. 8. Query: To utter a question, inquiry or a doubt, and 9. Recommend: To counsel or advise that something be done to solve the problem. Meta-actions describe or refer to any of the basic cognitive actions. They are usually statements of the participant when he is thinking about the problemsolving process but not about specific problem details. Eighteen distinct actions (9 actions and 9 meta-actions) are used to encode, at a prepositional level, the thoughts expressed by each participant during the problem representation and solving process. Independent coding can be carried out and the inter-coder reliability verified. 229 3. Data analysis The preliminary analysis of the protocols can be carried out in three stages: 1. Analysis of task-independent cognitive processes to determine general problem-solving strategies. 2. Task specific analysis for insights into how individuals solve a specific problem, and 3. Synthesis of the analyses to examine how general problem-solving strategies affect problem representation and solution outcomes. 3.1. Task independent analysis Each protocol can be encoded using software to facilitate the process. Protocol Analyst's Workbench (PAW; Fisher, 1991) is one such software tool that is independent of the problem domain. The interactive feature of PAW facilitates encoding that helps preserves the structure and content of the original transcript. PAW creates a matrix for each individual showing the frequency of use of each action in the protocol, and a table of percentages of each action used while solving the problem. Analysis of these percentages indicates the relative importance of different actions. PAW also gives the frequency of transitional and reflexive ties. Transitional ties occur when one action precedes another, reflecting the relations between the cognitive actions used to solve the problem. Thus if the action 'infer' is followed by the action 'recommend' it is recorded as a transition from the one to the other action. Reflexive ties occur when one action is followed by the same action, an example of which would be one 'infer* statement followed by another 'infer' statement where both statements are substantially different in content. Transitional and reflexive ties enable us to assess an action's weight in a flow model of the problem representation and solution process. They can also be used to set a quantitative threshold for considering a tie to be part of a cognitive process, to eliminate noise. The weights or percentages of these ties for each individual's protocol can then be entered in matrix form in any matrix manipulation software such as UCINET (Borgatti. Everett and Freeman, 1992) to create an individual matrix showing the percentage of transitional and reflexive ties on actions for each individual. Each matrix so obtained can be tested for similarity with the matrices of other individuals, using the correlation between an individual's matrix, and all the other individual matrices. This gives a data matrix showing the correlation between each individual and every other individual. This data matrix measures individual similarity or differences in transitional and reflexive ties. In order to test whether the results obtained are consistent with our hypothesized differences in characteristics between individuals, the data matrix can then be compared for similarity with a characteristic structure matrix. This 230 matrix has cells with 1 's if the pair of individuals share a similar characteristic and O's if they do not. In making statistical comparisons, structural autocorrelation may occur in the data matrix, due to the dyadic nature of relationships between individuals and non-independence of observations, leading to biased results. However Quadratic Assignment Procedure or QAP 1' 2, a nonparametric method of statistical analysis available in UCINET has been shown to give relatively unbiased results in such situations (Krackhardt, 1988). Other characteristics might also coincidentally influence differences among individuals in their problem solving strategies as represented by the data matrix. In such a situation QAP can be used to compare3 the similarity between the data matrix and the new characteristic structure matrix. The magnitude of the correlation coefficient between the two matrices and the level of significance would then indicate mutual influence. On the other hand the factors we hypothesize as influencing problem solving strategies may be correlated. QAP can then be used to determine the collinearitv between the two characteristic structure matrices. If this collinearity is severe, we can separate out the relative effects of both characteristics on the observed differences in problem-solving strategies between individuals. QAP multiple regression analysis4 can be performed using the data matrix as the dependent matrix and the two characteristic structure matrices as independent matrices. Having determined strategies used by individuals with specific characteristics or belonging to certain groups we can ask: Can these specific strategies be identified? We found helpful a graphical method for visualizing some details of the problem-solving strategies adopted by professionals. To make graphical comparisons, KrackPlot (Krackhardt et al., 1994) can display transitional and reflexive ties and the centrality and proportion of these ties for each action. A flow diagram displays the sequence of actions each individual follows while solving the problem. We found it appropriate5 to draw individual flow diagrams that showed only those sequential pairs of successive actions with transitional and reflexive ties that were common to individuals within a specified group. Our next step is to understand how these general problem-solving strategies could influence how individuals actually go about solving a complex and illstructured problem. The next two sub-sections indicate how general problemsolving strategies might influence the task specific outcomes and recommendations given by individuals. 3.2. Task specific process analysis Task specific process analysis requires an understanding of the domain knowledge that professionals use in problem solving. It asks whether professionals differ in their use of information to solve complex ill-structured problems. Problem behavior graphs (Newell and Simon, 1972) display the sequence of actions within a protocol and changes in representation of problem states in the individual's mind. A careful examination of the content of these graphs reveals 231 similarities and differences in how individuals belonging to similar and different groups solve problems. 3.3. Synthesis of results The most critical section of the analysis is the synthesis of what we know about the general problem-solving strategies used by individuals, and the specific outcomes of the problem-solving process. If done appropriately this synthesis can determine whether general problem-solving strategies play any part in determining problem representation and solution process and the nature of the solutions offered. A successful synthesis relates what we know about individual or group characteristics to specific strategies used by that individual or group. We will demonstrate the use of this method in the next section, using an example of the analysis of strategies in a complex and ill-structured problem. 4. Illustrative example: Policy problems as complex and ill-structured problems The problem of hunger in a country was chosen as an example. The problem of hunger meets the criteria of complexity and ill structuredness in many ways, so that individuals could use both domain-specific knowledge and general problem-solving strategies to solve the problem. There could be a wide variation in the knowledge each individual had or applied to the problem, derived both from information provided in the problem statement, and from external knowledge. 4.1. Policy problem The policy problem given to the participants consisted of the role to be played by the problem-solver followed by a brief description of the problem: As a senior policy maker in your country you are called upon to advise an international organization about what is needed to solve a specific policy problem in a country called Hungeria. Please explain what other information you might need to solve this problem and then what your recommendations would be? Brief description of the problem: Hungeria, with a population of 26 million, has nearly four million people who exist below the poverty line and around one million people who are undernourished and hungry because they do not have sufficient food to eat. In 232 estimating the numbers of people going hungry the Food and Agriculture Organization of the United Nations (FAO) uses as its criterion the energy intake level at which a person can barely survive which is a daily calorie intake below 1.2 basal metabolic rate (around 2100 Calories). Hungeria has a Gross Domestic Product (GDP) of $638 billion and is a net food exporting country. The birth rate in Hungeria is 15 per thousand and the death rate is 8 per thousand. 4.2. Specifying the complexity and ill-structuredness of the problem The problem of hunger frequently arises from complex linkages between the production, distribution and consumption of food within the context of political, social and economic institutions (Timmer, 1983). Lack of income and longterm employment can lead to hunger even in relatively developed countries. The food policy of a country is also constrained by the international economy, through its impact on balance of payments, foreign exchange rates and the import and export of food commodities. Instability in international food prices can seriously affect domestic food policies. Some of the actions that may help address the problem of hunger are: 1. Redistribution of assets: Highly unequal ownership of assets frequently affects food production and consumption. Land reform may achieve a redistribution of assets, as may taxation of the rich and subsidies to the poor. 2. Growth in income for the poor: Growth strategies may provide an increase in real incomes for the poor by providing new employment opportunities. 3. Reduced discrimination against certain social or demographic groups: Job opportunities may be improved for specific social or demographic groups by reducing discrimination against them. 4. Increase in the production offood: An increase in the production of food by a country may be achieved through technical change. The problem of hunger in a country therefore meets the criteria of complexity specified earlier. In order to ensure that it was also ill structured, the problem was designed with some 'contradictions'. This would induce multiple problem representations both in terms of problem structure and domain knowledge. The first "contradiction" was in the facts that Hungeria had a Gross Domestic Product (GDP) of S638 billion and a population of 26 million. This works out to a per capita GDP of around 524,000, among the highest in the world. Another 'contradiction' was the statement that Hungeria was a net food exporting country, implying that food production was not a serious problem. A third 'contradiction' was in the rather moderate birth rate and death rate, which were representative of a highly developed country. Fourth, no information was given about the occurrence of any natural disaster that could have caused the prob- 233 lem. These 'contradictions' eliminated various potential caus es of hunger in the country. A good solution would require that professionals either question that information or make suitable assumptions to assure cons istency before offering recommendations. The information provided in the prob lem was sufficient to support recommendations consistent with the 'contradictio ns', along two directions: 1. Targeted food subsidies could provide immediate relie f to the hungry (e.g. food stamp programs, monetary support programs and food-for-work programs). Given the high per capita GDP, this was an appropriate solution of the problem in the short run. 2. Long-term recommendations might include redistrib ution of income through tax policies or through productive jobs for peop le below the poverty line. 4.3. Data collection, coding and verification The professions chosen were architecture, medicine, law and engineering. Each of the participants6 held undergraduate degrees in these four fields. Each participant J was given 30-45 minutes to solve the problem while thinking aloud. The coding was carried out according to the methodol ogy described earlier (9 basic and 9 meta-actions). To test reliability of this coding, independent coding was carried out on a random sample (20%) of the encoded actions. Intercoder reliability for this sample was 96%. Finally Prot ocol Analysts Workbench (PAW) was used to encode each protocol 7 systemati cally. 4.4. Task independent analysis PAW was used to generate the percentages of actions used by each professional. The results are shown in Table 1. The percentages in Table 1 indicate that professionals used substantially more basic actions than meta -actions. However, architects were an exception among the professional grou ps in their rather extensive use of meta-actions. PAW was then used to analyze the relative proportions of transitional and reflexive ties. With a total of 18 types of actions (9 basic, 9 meta) for each professional we obtained a 18 x 18 square matrix like that in Table 2. Each matrix was then tested for similarity with the matrice s of all the other professionals using UCINET. This produced an 8 x 8 data matrix of the correlation coefficients as displayed in Table 3. Table 3 show s that individuals belonging to the same profession have higher correlation coefficients in general than individuals belonging to dissimilar professions, with two exceptions. Engineer 2 (E2) shows a higher correlation with Lawyer 1 (LI) and Lawyer 2 (L2) than he does with Engineer 1 while Physician 2 (P2) shows a higher 234 Table I. Relative importance of actions used by professionals (in %). Actions Assume Calculate Evaluate Infer Know Query Read Recall Recommend M-Assume M-Calculate M-Evaluate M-Infer M-Know M-Query M-Read M-Recall M-Recommend Participants Architect 1 Architect 2 Physician 1 Physician 2 Engineer 1 Engineer 2 Lawyer 1 Lawyer 2.9 1.5 13.2 10.3 16.2 19.1 6.6 3.9 14.5 2.6 6.6 17.1 1.3 2.6 13.2 3.9 5.3 5.3 11.8 0.0 3.9 1.3 0.0 0.0 3.3 3.3 0.0 13.3 6.7 13.3 21.7 18.3 8.3 0.0 0.0 1.7 5.0 0.0 3.3 0.0 0.0 1.7 3.2 1.6 11.3 4.8 8.1 25.8 12.9 9.7 4.8 0.0 0.0 0.0 12.9 0.0 1.6 0.0 0.0 3.2 12.2 0.0 9.8 17.1 4.9 22.0 0.0 2.4 19.5 0.0 0.0 4.9 2.4 0.0 2.4 0.0 0.0 2.4 7.8 3.1 4.7 20.3 14.1 6.3 10.9 0.0 23.4 0.0 4.7 3.1 1.6 0.0 0.0 0.0 0.0 0.0 2.5 1.9 15.4 16.7 19.8 0.6 3.1 6.8 20.4 0.0 0.0 3.7 3.7 0.0 1.2 0.0 0.6 3.7 14.9 0.0 2.1 6.4 36.2 5.3 0.0 1-1 28.7 0.0 l.l 3.2 I.I 0.0 0.0 0.0 0.0 0.0 4.4 5.9 5.9 0.0 4.4 4.4 7.4 0.0 0.0 0.0 0.0 4.4 Total meta-actions 20.6 31.6 11.7 17.7 12.2 9.4 13.0 5.3 Total basic actions 79.4 68.4 88.3 82.3 87.8 90.6 87.0 94.7 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 Total correlation with Architect I (Al) and Architect 2 (A2) than with Physician 1 (PI). In order to test whether the results were consistent with differences in professional education among the participants, the data matrix was compared for similarity with a profession structure matrix (Table 4), which had entries of 1 if a pair of participants belonged to the same profession and 0 if they belonged to different professions. QAP was used to calculate the correlation coefficient between the data matrix and the profession structure matrix. Table 5 shows that the correlation was 0.37. The probability of a correlation as large or larger than 0.37 with a random matrix was 1.1% indicating that the correlation was significant at the 0.05 level. As our sample included professionals who had obtained their professional education outside the U.S., these differences between professions might be due to other cultural differences. In order to examine the significance of such differences, a country structure matrix was created (Table 6) that had 1 's if the pair of participants were educated in the same country and O's if they were educated in different countries. QAP was used to compare the similarity between the data matrix and the country structure matrix. Table 7 shows that the observed correlation between the two matrices was 0.26. The percentage of random Recom Recal Read Query M-Recal M-Recom M-Que M-Rea M-F.va M-Inf M-Kno M-Cal AsMim ('alcu F.valu Infer Know M-Ass Calc 3 2 1 Eval 2 1 1 Infer 2 2 4 Know 2 M-Eva | | ----2 - .. 1 M-Inf • M-Reca M-Reco Query - | 1 1 I 9 7 II 1 0 .1 .1 4 21111 2 0 0 0 0 3 Recal 14 3 4 4 68 24 4 13 3 I 0 13 o 0 3 5 o 3 I I I | 12 I 9 7 II 0 Recom I 6 _._...-_..._... __ ........_.... ..(....... Read 212 --.-- M-Rea ......... M-Know M-Qne 12 ....... | 2 I ..._.......... III I M-Cal .. ..... M-Ass ..._...- „ Assu 7'dhfi' 2. Transitional and reflexive ties matrix for Architect I. 236 Table 3. Data matrix. 1 2 3 4 5 6 7 8 Al A2 PI P2 El E2 LI L2 1 Al 2 A2 3 PI 4 P2 5 El 6 E2 7 LI 8 L2 1.00 0.55 1.00 0.15 0.0? 1.00 0.44 O.J8 0.30 1.00 0.43 0.45 0.14 0.30 1.00 0.38 0.30 0.25 0.13 0.42 1.00 0.38 0.23 0.17 0.14 0.36 0.68 1.00 0.36 0.27 0.19 0.09 0.24 0.63 0.62 1.00 Bold - significant' at 0.01 level. Bold italics - significant at 0.05 level. Italics - significant at 0.10 level. Table 4. Profession structure matrix. 1 2 3 4 5 6 7 8 Al A2 PI P2 El E2 LI L2 1 Al 2 A2 3 PI 4 P2 5 El 6 E2 7 LI 8 L2 1 1 I 1 0 0 1 1 0 0 1 1 0 0 0 0 1 1 0 0 0 0 1 I 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 8 L2 Table 5. QAP correlation between data matrix and profession structure matrix. Observed value Average Standard deviation Proportion as large Proportion as small Correlation Matches 0.372 0.003 0.178 0.011 1.000 0.000 0.000 0.000 1.000 1.000 Table 6. Country structure matrix. 1 Al 1 2 3 4 5 6 7 8 Al A2 PI P2 El E2 LI L2 2 A2 1 0 0 1 0 0 0 0 0 1 0 0 0-0 0 0 3 PI 4 P2 5 El 6 E2 7 LI 0 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 M) 0 0 1 1 237 Table 7. QAP correlation between data matrix and country- structure matrix. Observed value Average Standard deviation Proportion as large Proportion as small Correlation Matches 0.258 0.002 0.175 0.102 0.899 0.000 0.000 0.000 1.000 1.000 Table 8. QAP correlation between profession structure matrix and country structure matrix. Observed value Average Standard deviation Proportion as large Proportion as small Correlation Matches 0.519 0.001 0.210 0.070 0.995 0.893 0.781 0.045 0.070 0.995 Table 9. QAP regression of data matrix on profession structure matrix and country structure matrix. Independent Intercept Profession Country Regression coefficients Un-stdized coefficient Proportion as large Proportion as small 0.28 0.14 0.05 0.994 0.082 0.293 0.006 0.918 0.707 /^-square: 0.144. Probability: 0.077. correlation's as large or larger than 0.26 was 10.2% indicating a correlation between the data and the country structure matrix lower than that between the data and profession structure matrix, and not statistically significant at the 0.10 level. The profession and country structure matrices had a fairly high level of collinearity. The QAP correlation coefficient between the two matrices was 0.52, which was significant at the 0. 10 level as shown in Table 8. In order to separate out the relative significance of professional education and country effects on the general problem-solving strategies used by professionals, QAP multiple regression analysis was performed using the data matrix as the dependent matrix and the profession structure and country structure matrices as independent matrices. Table 9 shows the results of the QAP regression. Profession appeared significant at the 0.10 level while country is not significant. Therefore there is evidence of profession-specific general problem-solving 238 strategies, reflected in the transitional and reflexive ties between actions, that are significantly similar for professionals belonging to the same profession and differ across individuals belonging to different professions. We then used the graphical method recommended earlier to view and com pare these general problem solving strategies. Shown below are the flow dia grams created using Krackplot (Figures 1 and 2) for two professional groups: engineers and lawyers in our sample, along with our interpretation. The common actions for engineers (shown in Figure 1) consisted of 5 out of the 8 actions and no meta-actions. These 5 actions account for 81% of all actions for Engineer 1 and 66% for Engineer 2. The actions Recommend, Infer; Assume and Query also show reflexive ties indicating the repeated use of these actions in solving the problem. No action appears to be truly central to the process for either engineer. The common actions for lawyers (shown in Figure 2) consisted of 6 out of the 8 actions and no meta-actions. These 6 actions account for 77% of all actions for Lawyer 1 and 94% for Lawyer 2. The actions Recommend and Assume also show reflexive ties indicating the repeated use of these actions in solving the problem. On the whole the action Know appears to be central to the process with the most cyclic ties to other actions. ( Interpretation The lawyers and engineers tend to use a large proportion of Recommend actions as compared to other professionals. However the process through which they arrive at a Recommend action differs substantially. Lawyers show a transition from actions Know and Infer to Recommend, while engineers showed transitions from the actions Assume, Query and Evaluate to Recom mend. Another difference is that transitions between Assume and Infer, appear for lawyers but not for engineers, although engineers use both these actions more than lawyers do. 4.5. Task specific process analysis Task specific process analysis relies upon the domain knowledge that profes sionals use. Problem behavior graphs were built to display the sequence of actions and changes in representation of problem states in the protocols for both physicians (extracts shown below), followed by our interpretation of the graphs. 239 Engineer 1 H-QUERY H-INFER 5 ' M-CMJCULMlf. \ M-EIM1.1MTE in Engineer 2 M-QUERY H-1HFER H-CALJCUUATE KNOW H-EUM.IMTE 240 Lawyer 1 H-qUERV H-INFER «-CflLCUU»TE M-EUM-IMTE EUM-UATE QUERY Lawyer 2 H-QUERY H-CM.CULATE H-IMFER 17 ft-EUMLlMTE EUM.UATE QUERV Fig. 2. 241 Extracts from problem behavior graphs: Physicians Physician 1 Physician 2 46. Inferi people are going hungry and people are undernourished 47. Inter: they are not carrying 2100 calories SO. »«i leanenil; food should be given no the people rather than exported 8. Infer: one million people fall in this category 9. Infer: the cut-off level is 2100 calories 25. Calculate: per capita income-24.DOC dollars per year 52. Recoeaendi foods costs should be low 54. XeeoBeead: production in the country 56. Kecaeviend: population control 60. » !. i emend; increase infrastructure of country 26. Know: that is the per capita income for the U.S.A 27. S-rtluatei how car. that be 28. Bvmloate: Not possible 29. Bymluee: figures are grossly inadequate to offer any recommendations 54. Bmloatei birth rate:seems to be a pretty stable population 55. Evaluate: 15 per thousand birth and 8 per thousand death rate is fine 56. Infer: not a rapidly growing population 61. Met«-»»i leeMiiil: not enough data tc sake serious recommendations Population control, which has medical implications, was considered by both physicians during the problem-solving process, with Physician 1 recommend ing population control as a solution. Physician 2 however critically considered the information given in the problem about the birth and death rates in the count ry and correctly inferred that population control was not the problem. They were also similar in their attempt to focus on diagnosis of the problem with an inability to diagnose due to the 'contradictions' (especially Physician 2, state ments 29 and 61). 4.6. Synthesis of results Do general problem solving strategies adopted by professionals play any part in determining the problem representation and solving process and the nature of solutions offered? Given below are some examples of use of these strategies by two professional groups: architects and lawyers. Architects The professional education architects receive trains them in architectural de sign (Akin, 1986: pp. 176-177). The design process shares with policy problems a complex and ill structured nature. Both architects sought a solution using strategies of architectural design. One of the dominant strategies was the Query Evaluate strategy shown below. 242 Query <==> Evaluate strategy I , i j Architect 1 Architect 2 13. Kr*laat«i unfair distribution of income 14. Query: income distribution graph? 15. Query: land use details of country? 16. Query: potential for country to grow more food? 17. Query: food eaten by the people? 34. >Md: the country is net food exporting 35. Infer: no real shortage of food in country. 36. Infer: food goes into the wrong channels 43. Queer: information on the administration, bureaucracy and politicians in the country? 44. Query: political structure in the country? 45. XnlumCet futile to draft policy that would not be implemented by bureaucrats 54. Infer: not a nourishment problem 23. Query: am I supposed to feed one million people? 55. evaluate: the problem is probably deeper than that. 58. X-Heoaemend: solution would have to be discussed with other people. 24. Xvaluate: the best vay -o feed 1 million people 29. XeceaBtnd: feed the undernourished 48. Xraluate: hard to say what types of job creation 49. Query: skills of population? 50. Query: education levels of population? 51. Query: climate of country? 52. Query: geographic location? 53. Query: food consumption habits? 61. JU « : retraining issue-people are not properly trained for available jobs ml retraining 62. 1Ui i 63. »«iu»ei people are not properly equipped for current job opportunities basic training. 64.Recommend: Thus Architect 1 used this strategy to determine that the problem was a wealth distribution problem for which he needed more information. He queried land use facts and food production facts that were not apt in the context of this problem, as the country had net food exports. He however heeded this point later on, which led him to realize the complexity of the problem (statement 55) and to offer a meta-recommendation (statement 58) rather than a recommen dation. Similarly Architect 2 used this strategy to make a short-term recom mendation of feeding the undernourished (statement 29). However his long term recommendations for creating jobs were restricted by insufficient infor mation (statements 49 to 53), and his consequent recommendations for creat ing jobs were based upon assumptions (statements 61 and 63) he made about the skills of the people. Lawyers Lawyers are trained to choose between alternative views, while focussing on facts that support their view (Wrightsman, 1987). The nature of their training predisposes them to focus on conflict resolution rather than on discovering the truth. Some elements of this professional training are reflected in the use of the actions Know and Recommend in their problem-solving strategies. Their primary problem-solving strategy used was the Know <=> Recommend strategy with reflexive ties on the actions Know and Recommend as shown below: 243 Know <=> Recommend strategy Lawyer 1 63 . tacoBaradi has to be a program for producers 64. Know: World Bank won't support anything against international or open markets 65. Knew: World Bank not a supporter of subsidies 66. »«i iiiaiaml: a prograir. for auto-consumption 67. RaoooMad: program tc increase productivity of land Lawyer 2 21. Know: wealth accumulation in small coranunities leads to some pressure for redistribution 28. B»i miMiiil provide people with education of practical skills 29. Know: this can be cheap though highly effective 41. Knowi without businesses, principles alone cannot sustain people 42. Jn n»«in1 use joint ventures to add value to agricultural products The strong Know <=> Recommend transitional tie for both lawyers appears to assist them to make recommendations. Thus Lawyer I uses his knowledge of the funding norms of the World Bank (statement 64) to foresee a problem for a short-term program of auto-consumption (statement 66). Similarly Lawyer 2 uses his knowledge of how hungry people behave (statements 24 through 27) to recommend education in practical skills as a way of solving the problem (statement 28). 5. Conclusions Our examination of the protocols suggests that the use of general strategies appears to be sometimes helpful and sometimes harmful in enabling profes sionals to solve complex ill-structured problems. For example the use of the Know <=> Recommend strategy by the two lawyers in our sample hindered their ability to use information provided in the form of 'contradictions'. The essential issue here seems to be determining when some general cognitive strategies facilitate the problem solving process and when they hinder it. A question that has interested researchers for some time has been: Are there singular differences in the general problem-solving strategies of professionals solving complex and ill-structured problems and can these differences be iden tified and measured? The methodology and analysis of this research note appears to identify such differences and may be applied to other real-world public policy problems. Using verbal protocol analysis, the sophisticated software currently available to analyze large and complex protocols, and the statistical and graphical tools described in this research note, valuable insight could be gained into the decisions taken by professionals. 244 Acknowledgements The authors are grateful for the comments of Linda Babcock, Danny Fernandes, David Krackhardt, Denise Rousseau and the comments of two anonymous referees on earlier drafts of this paper. Stephanie Mathews and Zhaoli Rong provided invaluable research assistance. The usual disclaimers apply. Notes 1. The algorithm testing the similarity of matrices proceeds in two steps. In the first step it computes Pearson's correlation coefficient (as well as the simple Matching coefficient) between corresponding cells of the two matrices. In the second step, it randomly permutes rows and columns of one matrix and recomputes the correlation. The second step is carried out hundreds of times in order to compute the proportion of times that the correlation based on random permutations is equal to or larger than the observed correlation calculated in step 1. If the correlation is positive, a low percentage (<0.05) suggests that the observed similari ty (as indicated by the observed correlation) between the matrices is unlikely to have occurre d by chance (cf. UCINET IV Version 1.0 - Network Analysis Software: User's Guide by Borgatti, Everett and Freeman, 1992: pp. 127-128). 2. The comparisons ignore the diagonals of both matrices. This avoids an artificial similarity due to both matrices having all 1's in their diagonals. 3. See note 2. 4. QAP regression procedure performs a standard multiple regression across corresp onding cells of the dependent and independent matrices in the first step. In the second step, it randomly permutes rows and columns (together) of the dependent matrix repeatedly and recomp utes the regression, as described in Note 1. 5. Two factors motivated this decision. First, that this would act as an approximation to eliminat ing any noise or false sequences between or within actions, from the diagram. Second , we would have established that individuals show higher levels of similarity within a group. 6. All the participants were male. There was some variation in the nature of the professional education each had received subsequent to completing his undergraduate degree and their current occupation, although the basic categories were restricted to professionals who had undergraduate and graduate degrees in those specific fields. The exceptions were one lawyer who was presently in a graduate program in public policy (Lawyer 1) and one engineer, currently in a graduate program in industrial administration (Engineer 2). Both physicians were practicing at a local hospital while the remaining participants were in graduat e school related to their field of undergraduate study. The professionals differed in the country where they did their undergraduate studies. Both physicians (India), both lawyers (Mexico), one engineer (France) and one architect (Switzerland) had non-U.S. undergraduate degrees. 7. Encoded protocols available on request. References Akin, O. (1986). Psychology of Architectural Design. London: Pion Limited, pp. 176-177 . Altemeyer, B. (1966). 'Education in the arts and siences: Divergent paths,1 doctora l dissertation. Carnegie Mellon University. Amsel, E., R. Langer and L. Loutzenhiser (1991). 'Do lawyers reason differently from psycholo gists? A comparative design for studying expertise.' in R. Sternberg and P. Frensch, eds.. Complex Problem Solving-Principles and Mechanisms, Hillsdale, New Jersey: Lawrence Erlbaum Asso ciates, pp. 223-250. 245 Borgatti, E. and Freeman (1992). 'UCINET IV Version 1.0 - Network Analysis Software: User's guide'. Doktor, R. (1969). The development and mapping of certain cognitive styles of problem solving.' doctoral dissertation, Stanford University. Ericsson, A. K. and H. A. Simon (1983). Protocol Analysis: Verbal Reports as Data. Cambridge. MA: The MIT Press. Fisher, C. (1991). 'Protocol analyst's workbench: Design and evaluation of computer-aided protocol analysis.' doctoral dissertation. Carnegie Mellon University. Funke. J. (1991). 'Solving complex problems: Exploration and control of complex systems,' in R. Sternberg and P. Frensch, eds.. Complex Problem Solving - Principles and Mechanisms, Hillsdale. New Jersey: Lawrence Erlbaum Associates, pp. 185-222. Krackhardt. D.. J. Blythe and K. McGrath (1994). 'KrackPlot 3.0: An improved network drawing program.' Connections 17 (2): 53-55. Krackhardt. D. (1988). 'Predicting with networks: Nonparametric multiple regression analysis of dyadic data,' Social Networks 10, pp. 359-381. Newell, A. and H. A. Simon (1972). Human Problem Solving. Englewood Cliffs, N.J.: Prentice-Hall. Schwartz, S. and T. Griffen (1986). Medical Thinking - The Psychology of Medical Judgment and Decision-Making. New York: Springer Verlag. Simon. Herbert A. (1997). Administrative Behavior, 4th edition. New York: The Free Press. Simon. H.A (1973). The structure of ill-structured problems,' Artificial Intelligence 4, pp.181-201. Timmer, P., Walter F. and Scott Pearson (1983). Food Policy Analysis. Baltimore: The Johns Hopkins University Press. Voss, J., S.W. Tyler and L. A. Yengo (1983). 'Individual differences in the solving of social science problems,' in R. F. Dillon. R. R. Schmeck, eds., Individual Differences in Cognition, New York: Academic Press. Voss, J. F. and T.A. Post (1988). 'On the solving of ill-structured problems,' in M.T.H. Chi, R. Glaser, and M. J. Farr, eds., The Nature of Expertis, Hillsdale, N.J.: Lawrence Erlbaurn Asso ciates, pp. 261-285 Winter, D.G., D.C. McClelland and A. J. Stewart (1981). A New Case for the Liberal Arts. San Francisco, California: Jossey Bass Publishers. Wrightsman. L. S. (1987). Psychology and the Legal System, Monterey, California: Brooks/Cole Publishing Company.