Cognitive processing and knowledge representation in decision making under uncertainty1 John Fox2 and Richard Cooper3 Abstract This article is a contribution to the current debate on the role of cognitive theory in our understanding of human judgement and decision making under uncertainty. We argue, with Busemeyer et al and others, that the theoretical and methodological traditions of the JDM community and mainstream cognitive science are divergent to an undesirable extent, and that the exploitation of established concepts of information processing theories and knowledge representation would considerably strengthen the field. The paper revisits and extends an earlier study Making decisions under the influence of memory (Fox, 1980) in order to explore how these proposals might be applied in practice. A central technique developed by cognitive scientists is that of computational modelling; the paper makes extensive use of a new modelling tool, COGENT, to show how cognitive theory can significantly illuminate the mental processes involved in complex, real-world decisions. To appear in Scholz and Zimmer (eds) Qualitative theories of decision making, in preparation. 1 We would like to thank David Budescu, University of Illinois; Alastair McClelland, University College London; David Hardman of City University, London, and John Morton of the MRC Cognitive Development Unit, London, for helpful comments on an earlier draft of the chapter. 2 Imperial Cancer Research Fund, Lincoln’s Inn Fields, London 3 Department of Psychology, Birkbeck College, Malet Street, London Cognitive processing and knowledge representation in decision making under uncertainty John Fox and Richard Cooper Introduction The study of human decision making is an important topic in modern psychology. After all, our effectiveness as individuals, and as societies, is profoundly influenced by individual’s abilities to make the right choices when we are confronted by significant threats or opportunities. Considering the importance of decision making it is surprising that the field does not seem to have achieved the central position in cognitive research that other traditional psychological topics have, such as the study of memory, perception or language. Why is this? One possible reason is that the theoretical frameworks, experimental methods and even the sociology of the decision making research community differ significantly from most other areas of psychology and cognitive science. A notable feature of the field of judgement and decision making (JDM) is that it is highly multidisciplinary. Researchers in economics, management science, sociology and statistics are all contributors to JDM, as well as psychologists. From the resulting intellectual soup has emerged a preoccupation which is not common elsewhere in psychology. This is a concern with normativeness, or “the study of guidelines for right action” (Fishburn, 1988). Economists who are interested in consumer choice, for example, are often concerned with “rational” decision making. Statistical decision theorists formalise such ideas with normative concepts, such as expected utility, a measure based on mathematical probability and quantitative value scales. Social scientists and management scientists also frequently take normative economical and statistical frameworks as given, though their explanatory accounts tend to be more informal and intuitive. Psychologists who are interested in the processes of decision making typically see their role as illuminating and/or critiquing the application of normative thinking to the description of human judgement.Theories of judgement, for example, often adopt or adapt classical decision theory in order to model human decision functions (e.g the concepts of subjective probability and subjective expected utility). Studies of indidividual and organisational decision making often use the language of normative theory to describe decision making behaviour and its prescriptions provide the standards against which human performance is judged. Cognitive science, like the specific field of judgement and decision making, is concerned with the study of mental processes, but it has developed very different theoretical and explanatory frameworks. Cognitive scientists do not typically take the view that there is a normative theory of how we should remember, see or hear, for example, but rather that these are processes whose operation (good, bad or indifferent) is simply to be understood in terms of the mechanisms which implement them. While cognitive scientists often have very different interests and theoretical positions, they generally share an interest in “how things work”. Cognitive psychologists tyically try to understand the mental functions which underpin behaviour, and cognitive neuroscientists try to understand these functions in terms of their implementation by specialised structures in the brain. In contrast, as Jane Beattie of Sussex University has observed4, judgement researchers are “behaviourists”, being primarily concerned with what people do rather than how they do it. With few exceptions JDM researchers do not concern themselves, say, with the role of memory in decision making, or investigate the effects of brain damage or 4 Personal communication (1994) psychiatric dysfunction on choice behaviour, or explore uncertainty management in primate problem solving. This is not, in itself, a criticism of JDM research or its methods, but to outsiders like the authors it is a concern that there is such a discontinuity between this important subfield of psychology and other areas of cognitive science. In preparing this paper we have found that a number of writers who are prominent in the field have also expressed concern about the relatively poor links between theoretical research in JDM and mainstream cognitive psychology (e.g. Wallsten, 1980) yet “The few cases of intersection ... are overwhelmed by numerous other examples where research in one field proceeds without taking notice of findings from the other fields” (Busemeyer, 1995, p xii). Lola Lopes of the University of Iowa has observed that there are, roughly, three general theoretical approaches to understanding human decision making; algebraic, procedural and experiential approaches. Algebraic (also called “structural”) theories model judgment in terms of mathematical choice functions such as expected utility functions. The aim of these theories is to predict what people (students, consumers, managers, policy makers, doctors, patients, or even courts of law) will decide under what conditions. Lopes estimates that perhaps 95% of judgement research is carried out within this tradition which, unlike cognitive science, is relatively unconcerned with mental processes or how individual knowledge and experience may affect decisions and decision making strategies. A number of other writers have questioned the centrality of the normative approach to decision making under uncertainty from various points of view (e.g. Shanteau, 1987; Beach, 1990; Fox, 1990, 1994; Fox et al, 1990; Gigerenzer, 1994). However, these doubts continue to fail to take root. Perhaps this is because they are questioning an orthodoxy which is deeply rooted in modern political thought as well as in the dominant scientific tradition. The neglect of non-algebraic frameworks seems highly undesirable to us 5. Potentially important explanatory concepts from cognitive science which might have theoretical or practical value may be lost, and the possibility of synergy with research in other traditions missed. We guess that one of the reasons for this neglect is that many JDM researchers assume that the cognitive processes involved in everyday decision making are so complex as to make “mentalistic” theories intractable. We would question this. In his book Unified Theories of Cognition Allen Newell describes a general theory of cognitive processing which has at its heart a simple but powerful decision making process. While Newell’s theoretical programme is ambitious and has serious difficulties (e.g. Cooper and Shallice, 1995) the work suggests that decision processes, and many examples of complex behaviour which depend upon decison processes, can be successfully modelled using information processing and other cognitive concepts. This gives us a particular reason to share what appears to be a growing concern that judgement research is disconnected from mainstream cognitive research. In this article we have attempted to demonstrate a number of ways in which bridges might be built between the JDM and cognitive science communities, using information processing concepts and computer simulation techniques. Our general objectives are to set out a “mentalistic” account of decision making in complex decision making tasks. More specifically we aim to demonstrate that: 1. decison making can be productively viewed as a process of applying symbolic knowledge to information about a specific situation in order to resolve uncertainties in problem solving 5 Politically as well as scientifically! 2. computational modelling techniques can be used to precisely express psychological theories and to formulate predictions from them 3. the modelling techniques can be used to explore and evaluate competing theoretical claims We hope to show that there is considerable scope for applying recent developments in cognitive science to in order to understand human decision making. Adoption of these methods may improve the descriptive power of psychological decision theory, and establish stronger links with other areas of cognitive research. The starting point for our presentation is a study of medical decision making and a model of the functioning of memory mechanisms in the decision making process (Fox, 1980). Memory is crucial to most decision making yet has almost no place in modern judgement theory. As Weber et al (1995) comment, “most models of decision or judgement performance do not explicitly [incorporate] considerations of memory processes and [or] representations of information in memory”. Section 2 outlines the main features of the 1980 study and the data obtained from it, together with a comparison of the behaviour of a group of medical students carrying out a diagnosis task and the behaviour of a computer model of the role of memory processes in their decision making. In section 3 we present a reimplementation of this model using COGENT, a tool recently developed for modelling cognitive processes and investigating cognitive theory. COGENT allows us to explore a number of additional theoretical questions about knowledge representation and cognitive processing which were not possible in the earlier paper.. 2. Making decisions under the influence of memory Fox (1980) reported a study in which medical students learned to make diagnostic decisions in a realistic but carefully designed task environment (Figure 1). The students were required to diagnose “patients” simulated by a computer, in which the statistical relationships between diseases and symptoms were precisely known. During the experiment a number of aspects of the behaviour of the students were comprehensively recorded. At a number of points in the experimental procedure the subjects also took part in a memory retrieval task. This tested their growing knowledge of the relationships between the diseases and symptoms, acquired as a result of their increasing experience on the task. The data acquired during this study were used to inform the development of a model of the medical students’ decision making process. This model was presented in terms of an interaction between cognitive processing mechanisms and the representation of medical knowledge in memory. The 1980 paper also reported on a number of computer simulations of the students’ performance. The aim of the simulations was to investigate whether (a) it was possible to predict features of the students’ decision making and (b) how the theoretical assumptions of the model could impact on the accuracy of the predictions. Figure 1: The task environment described by Fox (1980). On the left is a computer display which was used to simulate patients, and the inset panel was the control panel used to select questions and enter diagnoses. On the right is a computer terminal which was used for the memory experiment (see text). Laboratory task and data collection The study was organised as a number of blocks of trials, on each of which a patient was simulated by a computer, and the subject’s task was to decide the diagnosis. On each trial the computer presented one of five symptoms: dysphagia (difficulty in swallowing); vomiting, headache, earache and pyrexia (raised temperature). The students knew that the “patient” might be suffering from any one of five diseases: tonsillitis; laryngitis; meningitis; hepatitis or (infection of the parotid glands). After being told the presenting symptom the subject could ask about the presence or absence of any of the other symptoms, in any order, and could offer a diagnosis at any point. The selection of the presenting symptom, and the answers to any questions which were asked, were determined by the computer by reference to a set of conditional probabilities (reproduced in table 1). The experiment consisted of 4 blocks of 25 trials in which the subject was required to diagnose what was wrong with each patient. Each block of diagnosis trials consisted of 5 instances of each of the 5 diseases (in randomised order). Various features of the students’ decision making were recorded by the experimental equipment. These were (1) the number of questions asked for each decision, (2) the order in which questions were asked, and (3) the diagnosis that the student arrived at on each trial. Dysphagia Tonsillitis Laryngitis Meningitis Hepatitis Parotitis 1.0 0 1.0 1.0 1.0 Vomiting 1.0 0 0.50 0 0.25 Headache 0 0.50 0.75 0.50 0.75 Earache 0 1.0 0 0 0.25 Pyrexia 0 0 0.25 1.0 0.25 Table 1: The conditional probabilities relating diseases and symptoms in the patient simulation. Figure 2 shows the overall pattern of questioning shown by the medical students during the final block of trials, for each presenting symptom. The first node in each tree is the presenting symptom, and the branches show the main “routes” the subjects took in their question selection. The numbers in the figure are the overall frequencies of the most common routes. While there is considerable variety in the patterns there are also clear preferences in the question selections. Figure 2: The question selection patterns for each presenting symptom, recorded for 18 subjects in the final block of decision making trials. Each circle represents the presenting symptom or a question (Dysphagia, Vomiting, Pyrexia, Headache, Earache). A single line into a question means the previous symptom was always present or always absent while a double line means that the question was asked whether or not the previous question received a consistent answer from the computer. Small filled circles indicate the point at which a diagnosis was given and the numbers underneath indicate the frequency with which the particular path through the tree was taken. (For clarity all paths that were used only once during the experiment have been excluded.) Turning to the subjects decisions, all the students learned to do the task and to achieve a high level of performance over the four blocks; Table 2 summarises the data obtained during the fourth block of trials. Each set of diagnosis trials was interleaved with a memory task which was intended to test the students’ knowledge of the associations between the diseases and symptoms. The memory task consisted of 25 trials in which a proposition of the form <Symptom> is associated with <Disease> was presented to the subject. In each case the subject was asked to confirm or deny the statement presented, and their responses and response times were recorded. Figure 3 reproduces the group average response times for the 25 propositions. Although there is considerable scatter in the data it can be seen that there is a systematic relationship between the conditional probability of a symptom occurring with a disease and the response time; the closer the probability is to 0 or 1 the faster was the response to confirm or deny the association. Dysphagia Dysphagia Vomiting Headache Earache Pyrexia No question 2 3 38 10 0 48 8 33 9 21 31 0 X Vomiting 7 Headache 31 Earache 38 Pyrexia 24 X 22 X X 3 23 8 43 25 0 X 19 0 X Diagnostic accuracy by comparison with the true presenting disease: Average number of questions before giving a diagnosis: 81 % 2.12 Table 2: Summary of the main results of the medical students’ behaviour on the last block of trials. The table shows the “one-ply” data, the frequency with which each question was asked given each presenting symptom (the X shows the most frequent question). Figure 3: the overall response times for subjects confirming or disconfirming the relationship between symptoms and diseases, as a function of their actual frequency of co-occurrence (probability). Simulation experiments and data collection The principle aims of the simulations were to compare the predictive or explanatory power of a family of algebraic models of the students’ performance with alternative “cognitive” models. The basic algebraic model consisted of a number of quantitative functions: (1) for revising probabilities in the light of symptom data (Bayes’ rule); (2) for evaluating the expected information yield of alternative questions (based on a probabilistic entropy maximisation function); (3) a decision function which selected a diagnosis based on the posterior probability of the diseases given the available symptom information (the simulation would decide that a disease Di was the diagnosis if the probability of Di given a set of symptoms Ej exceeded some threshold parameter Theta). Table 3 summarises the results obtained by running this simulation. These are to be compared with the summary data shown in Table 2. The predictive power of the model was judged reasonable though not convincing. A number of variants of the model were developed (e.g. by varying the hypothetical relationship between the students’ subjective probabilities corresponding to the objective probabilities shown in table 1) but none of the manipulations resulted in a significant improvment in the predictions of the model. Dysphagia Vomiting Headache Earache Pyrexia No question Dysphagia X Vomiting X Headache X Earache X Pyrexia X Average number of questions before giving a diagnosis: 1.57 Table 3: Summary of the main results of the Bayesian decision model. The table shows the “one-ply” data, the frequency with which each question was asked given each presenting symptom (the X shows the most frequent question). The second simulation consisted of an information processing architecture with two main components; a reasoning mechanism and a working memory. The former was used to implement a decision process as a set of if...then... production rules embodying knowledge about the medical task; the rules reacted and progressively added to the contents of working memory. As data entered working memory rules could “fire” if particular data patterns became true, and conclusions were then added to working memory as new data. Each addition might result in further rules firing, so that the reasoning and decision making process was modelled as a cyclical process. The rules making up the decision process included rules for generating diagnostic hypotheses (in response to symptom data); anticipations (of whether symptoms would be present or absent); and queries (about the presence of particular symptoms). Figure 4 reproduces the rule set published in the original paper. The central point to note is that in this model all the rules are qualitative There is no quantitative representation of uncertainty, such as the probabilities of diseases or the conditional probabilities of diseases given symptoms. Uncertainty is represented implicitly, in the availability of knowledge in memory. The basis for assigning relative availability to information retrieved into working memory from the knowledge base was the average response times of the subjects in the memory task (Figure 3). Put simply, the model assumed that the knowledge which “comes most easily to mind” (that is, becomes available in working memory) will have a greater influence on the processes of reasoning and decision making than knowledge which becomes available more slowly. The availability of information was encoded in the order of the conclusions of the rules (Figure 4). The effect of this ordering is to determine the order in which other rules will fire. This will control the order in which hypotheses and anticipations are added to working memory, and hence how the rules which produce queries (and hence questions) are instantiated. In short, if memory is systematically biased to retrieve some information more quickly than others then this information will tend to dominate reasoning and decision making. Figure 4: The production rules of the decision procedure in simulation 2 reproduced from Fox (1980) The results of running two rule-based simulations are shown in table 4, to be compared with the subject data in table 2 and the results of the algebraic model in table 3. It will be seen that the behaviour of these two simulations resembles that of the subjects more closely than the behaviour of the quantitative models. The conclusions from this simulation study were that a qualitative representation of task knowledge could implement a decision procedure whose decision making performance was more similar to that of the human subjects than a variety of quantitative models, and that the execution of this decision procedure, in interaction with a memory retrieval process, yielded a better account of the sequential features of the subjects’ question selection behaviour than functions designed to maximise information gain. To summarise, the qualitative decision model was able to predict interesting features of behaviour on a relatively complex task, and did this within a theoretical framework which was broadly in line with fairly established ideas about the organisation of human cognitive processes. Dysphagia Vomiting Headache Earache Pyrexia No question Dysphagia D V Vomiting DV Headache D V Earache D V Pyrexia DV Average number of questions before giving a diagnosis: 1.73 Table 4: Summary of the main results of the qualitative models. The table shows the “one-ply” data, the frequency with which each question was asked given each presenting symptom (the D shows the preferred question when the DISCRIM rule was used, and the V shows the preferred question when the VERIFY rule was used to select the question). In the remainder of this paper we present some new results from additional simulations. The motivation for the additional modelling efforts is to explore these conclusions in more detail, and to illustrate how recent developments in computational modelling may be able to provide further insights into the cognitive processes involved in decision making. 3. How critical are processing and representational assumptions to the decision making model? In assessing the value of a computational model such as that above, two questions are likely to come to mind. First, how much of a model’s explanatory power comes from theoretically significant assumptions about information processing and/or knowledge representation, as distinct from purely technical features of the program? Experiments 1 and 3 explore this question.(This is a general methodological problem in simulation work which is extensively discussed by Cooper, Fox, Farringdon and Shallice, 1996.) Second, we may ask the complementary question: if we change the model in such a way that important theoretical assumptions are violated, will the behaviour expressed by the model also change so that it no longer provides as good an account of subjects’ behaviour? This is the focus of Experiment 2. The “simulation experiments” described here have been carried out using a new tool for cognitive research, the COGENT cognitive modelling package. COGENT is designed to make it relatively easy for the psychologist to explore theories of cognitive processing by implementing a computer simulation of the theory, and systematically experimenting with alternative theoretical assumption in order to formulate and test empirical predictions. This can be difficult to establish by hand if we are working with complex models and/or complex tasks. COGENT provides a kind of construction kit, a set of standard software modules that can be linked together and parameterised in various ways in order to construct any of an indefinitely large set of specific models. Once constructed the model can be run like any computer simulation in order to observe its behaviour and compare it with human behaviour. The main modules currently provided by COGENT are: storage buffers, symbol processing modules, connectionist (subsymbolic) networks and data transmission functions (links between modules). A COGENT buffer is simply a place to store items of static information (such as chunks of knowledge about diseases and symptoms) or dynamic information (such as inferences about a patient). A COGENT process is some sort of information processing mechanism, which is used to manipulate (add, delete or modify) the contents of buffers to which it is linked. We shall describe several variants of the decision model reported by Fox (1980) which have been built using the COGENT system. Technically COGENT is very different from the simulation software developed for the 1980 studies, but the empirical evaluations are carried out using the same published data. Experiment 1: Changing the implementation details Figure 5 shows a COGENT implementation of the 1980 decision model. The model is organised into two levels, illustrated by the contents of the two windows in the figure. The first level (in the left window) is a simulation of the complete experimental situation shown in figure 1, i.e. a simulation of the student (subject model), and a separate but linked simulation of the task environment (the experimenter model). The experimenter model simulates the patients, and records and analyses the behaviour of the subject model. It communicates with the subject model as needed by sending messages down the link (“presenting” symptoms and answering questions). The subject model is shown in more detail in the right hand window. It consists of a working memory implemented as a buffer (shown as a rectangle with rounded ends) and a decision procedure implemented with a symbolic computation process (pointed ends). The second process in the right-hand window is not a significant part of our model, but only a mechanism whereby the decision process model interacts with the experimenter model. It does this by sending information to the experimenter model (as when it asks questions or gives its diagnosis) and receives messages from it (the presenting symptom or the answers to its questions). The COGENT approach to building cognitive models out of standard components frees the psychologist from a great deal of (potentially laborious) programming. However, these standard components may have properties which are not intended to be part of the model builder’s theory (Cooper et al, 1996). The structure of working memory in the 1980 simulation is a case in point. In the published simulation working memory was modelled as a hierarchy of chunks, each of which could be accessed through a header feature and then searched serially. This hierarchical data-structure could be implemented, but the standard COGENT buffer is primarily designed to store simple sets of elements (which can be accessed randomly or scanned with recency or primacy priority) reflecting common assumptions about working memory organisation. In the first simulation experiment we decided to reimplement the 1980 model using a standard COGENT buffer for the working memory. (Otherwise the decision process consists of a set of rules which is precisely equivalent to those in Figure 3, apart from minor syntactic differences.) Since the decision model makes no assumptions about the structure of working memory other than the order in which it is loaded with information (hypotheses, anticipations, queries etc), and the way in which this information is accessed, we should expect these implementation details to have no behavioural consequences. Figure 5: Reimplementation of the 1980 model. COGENT provides a facility to view any buffer continuoualy during the execution of a model. Figure 6 illustrates the set up of working memory, together with the contents of the buffer at the end of a typical diagnosis trial. At the top of Figure 6 is an area where various functional parameters of the buffer can be set. These provide a number of ways to specialise the standard buffer, to give it the properties required for a specific model. It can be seen, for example, that in order to use the buffer as a working memory for this model it has been set up for a particular mode of access (LIFO, lastin-first-out) and that its contents do not decay over time (so no information is lost during a trial). Obviously these parameters could be changed if a particular theory required different assumptions. Below the setup area is a window in which any initialisation data that are required on each trial can be specified (there are none in this case). In the bottom window we see the contents of the working memory at the end of a simulated diagnosis trial. The operation of the model is as follows. When a cogent model is executed all processes in the model are executed in parallel; reading and writing to any buffers to which they are connected, sending messages to other processes and so on. Although computation is parallel and distributed, however, data are added to working memory sequentially, causing rules to fire and send messages and/or update working memory on specific cycles. The numbers on the left of each working memory element show the cycle on which an element was added; it can be seen that information builds up cumulatively during the run. At the bottom of the buffer we see that the patient’s presenting symptom was (“told(vomiting, present”) on cycle 3, then two hypotheses, hepatitis and meningitis, are added on cycle 4. This is followed on cycle 5 by adding expectations; the symptoms associated with these two diseases. On cycle 6 the model generates a query about whether the symptom headache is present. (The model has been set up to be run in “verify” mode, in order to try to confirm the first hypothesis it finds in memory, which in this case is meningitis.) The answer comes back from the task simulator on cycle 9 (it takes a couple of cycles to output the question and get the answer through the input process). Given this new symptom information the model is able to make a diagnosis on cycle 10. (Note in passing that the model has also generated another query, about the presence of earache, on the same cycle. This is because of the parallel nature of the system, which allows inferences to occur asynchronously. However this query is not translated into an overt question since a diagnosis has already been arrived at.) Figure 6: State of working memory at the end of a typical diagnosis trial Table 6 shows the results of testing this new implementation of the 1980 model. If it is compared with the results reported in the original simulation (Table 4) it will be seen that the question selection pattern is identical. The quantitative results are also closely similar. (Actually the average number of questions predicted by the COGENT model is 2.1, which is closer to the equivalent figure (2.12) seen in the subject data in table 2 than the original model. However this is due to random variation in the simulated patient sequence and has no significance.) In short the differences in the implementation details appear to have had no detrimental effect on the fit between the model and the subject data. Dysphagia Vomiting Headache Earache Pyrexia No question Dysphagia D V Vomiting DV Headache D V Earache D V Pyrexia DV Diagnostic accuracy by comparison with the true presenting disease: D 72%; V 94%; overall 83% Average number of questions before giving a diagnosis: D 1.5; V 2.7; av = 2.1 Table 6: Results of the COGENT reimplementation of the 1980 model. The results are identical to those reported in the earlier implementation, which had a very close firt with the observed subject data Experiment 2: The effects of working memory access assumptions In contrast to technical changes which have no theoretical motivation, any change to the model which might affect the significant parameters of information processing would be expected to change behaviour. Trivially, if the rules in the decision procedure embodied no knowledge about tonsillitis, then the decision process could not form hypotheses about tonsillitis, select questions to verify tonsillitis, nor diagnose the condition. This expectation is easily confirmed by simulation. Of greater interest is the central theoretical claim of the 1980 paper, that the operational use of knowledge is significantly influenced by its availability in memory. If this is changed then, according to the theory, it should produce a significant change in the pattern of behaviour, notably to the order in which questions would be asked. In experiment 2 we investigated two changes to the effects of availability by changing the access mechanism of working memory. Recall that the 1980 model claims that working memory is loaded with information about hypotheses and expectations in an order which is determined by the reliability of the association between each symptom and disease. If we change the working memory access, so that this relationship is no longer mirrored in the availability of information in working memory, then we would expect the model to behave differently (though how differently is not immediately obvious in a model like the present one which has scope for considerable operational variability). The first experimental manipulation therefore consisted of changing the working memory access regime so that the items which were loaded first into working memory would now be the last to be available to the question selection process. The results of this variation are shown in Table 7. The fit with subject data has now dropped dramatically, from a perfect fit of 5 agreements to only 1 agreement for the and discrimination strategy. The fit for the verification strategy is equally poor. Dysphagia Vomiting Headache Earache Pyrexia No question Dysphagia DV Vomiting D V Headache DV Earache DV Pyrexia DV Diagnostic accuracy by comparison with the true presenting disease: D 80%; V 96%; overall 88% Average number of questions before giving a diagnosis: D 2.0; V 3.0; av = 2.5 Table 7: Results of rerunning the decision model with modified working memory access. Access is based on linear scanning, as in experiment 1, but the direction of scanning is reversed. Dysphagia Vomiting Headache 5 Dysphagia Earache Pyrexia No question 7 V D 7 Vomiting 2 4 D Headache 3 4 Earache 6 3 Pyrexia 2 1 V 4 V 1 4 D 7 4 5 V 3 D 1 4 V 4 D 2 5 2 V 2 3 1 1 D Diagnostic accuracy by comparison with the true presenting disease: Average number of questions before giving a diagnosis: D 2.3; 4 V D 80 %; V 92 %; overall 83% V 3.1; av = 2.7 Table 8: Effect of modifying the access of working memory from linear scanning to random selection. This destroys the availability effect and introduces considerable behavioural variety (as shown by the inset numbers which are the frequencies with which questions were selected by the discrimination and verification strategies (left and right, respectively). Little effect on diagnostic performance is expected, however. Since diagnosis is carried out by rules which recognise particular patterns of symptoms the diagnosis decision is primarily determined by the symptoms the patient has rather than the order in which the information is presented. Table 7 confirms this: the diagnostic performance is unaffected by the changed memory access. A second manipulation of the working memory model aimed at disrupting the relationship between symptom reliability and availability has also been carried out. This time working memory has been set up to be randomly accessed. Every time a rule in the decision procedure attempts to match one of its conditions with information in working memory a random procedure is used to select the item to be matched, rather than carrying out a linear search as in the previous experiments; COGENT will keep on randomly selecting items from the buffer (without replacement) until it finds a match or exhausts the buffer without finding a match. The consequence of this is to nullify any effect of loading working memory in any particular sequence. Table 8 shows the effect. Examination of the data reveals two obvious features: (1) the fit between the first question preferred by the model and that preferred by the subjects has again been reduced by comparison with the data in Figure 2. (2) the clear peaks in the orginal deterministic model have now been “smeared” out, with the result that there are a number of peaks. Given a particular presenting symptom, headache say, the model sometimes asks about vomiting first, sometimes earache first. Overall the most preferred questions no longer show a resemblance to the most preferred questions of the subjects. Although the memory access likelihood is equally distributed over the working memory by the COGENT access function, the questions are clearly not being selected randomly (some questions are never asked under some conditions). The reason for this is that availability of information in memory is not, in fact, the only factor which determines how decision making will proceed; there are also logical constraints built into the decision rules, and these will exclude certain questions for reasons which are due to the structure of the task as distinct from the organisation of the architecture. For example, since parotitis is never associated with vomiting the decision procedure will obviously never attempt to verify a diagnosis of parotitis by asking whether vomiting is present. Experiment 3: Equivalence of different knowledge representations In Experiment 1 we showed that changes to the representation of information in working memory had no effect on behaviour, so long as parameters which affect the interaction between the logical decision process and the accessibility of information are not changed. Experiment 2 showed the reverse situation; the representation can be left unchanged but if the processing parameters are significantly altered the decision making performance of the system is not significantly affected, though the order in which questions are asked is much changed. In neither experiment was the encoding of medical knowledge in the decision procedure significantly changed from that used in the original 1980 model (Figure 3). This raises the question of whether behaviour is sensitive to the representation of medical knowledge or not. In principle our answer to this question is the same as when we considered whether the encoding of information in working memory would affect behaviour; so long as we do not change the medical content of the knowledge base, nor modify critical execution parameters such as memory retrieval parameters, we would expect no change in the behaviour of the system. Figure 7: Abstraction of generalised diagnostic procedure from “special-case” medical knowledge. Figure 7 shows a revised COGENT model in which a generalised decision procedure for diagnostic decision making has been separated from specific knowledge about diseases and symptoms. This decision procedure can be viewed as a generalised strategy for proposing diagnostic hypotheses, anticipating symptoms and selecting questions which can be used to differentiate any set of alternative diagnoses. Figure 8 shows this generalised diagnostic strategy. The decision process now consists of only 5 rules, one for each distinct function (1. generate hypotheses; 2. generate anticipations; 3. select question to verify a hypothesis; 4. select question to differentiate two hypotheses; 5. make a diagnosis). Instead of having a large number of specialised medical rules in which the decision process is implicit, the decision process is now explicit. The the specialised domain knowledge is stored in a separate knowledge base, as a collection of facts like “tonsillitis is associated with dysphagia”. Logically the two models are exactly equivalent though there are a number of well known advantages to the latter kind of representation, including: Diagnostic expertise can be simply increased by learning additional facts, such as “peptic ulcer is associated with vomiting” or “chicken pox is associated with spots”. Once these facts are added to the knowledge base they can be used immediately without having to reprogram the decision procedure. The medical knowledge is represented “declaratively” rather than “procedurally”. In the first model the medical facts required to do the task are embedded in the rules which form the diagnosis procedure, so this knowledge cannot be used for any purpose other than diagnosis. When encoded as distinct facts, as in the second model, the knowledge can be reused for other purposes. Another use would be to introduce an additional process which would use the same medical knowledge base in order to explain a decision, for example. Figure 8: A first-order implementation of the decision procedure. Each rule is a “first-order” inference rule which is evaluated by matching the antecedent terms with the contents of an associated memory. Variables (capitalised words, such as Disease) are instantiated by the matching process, as when the Disease variable is instantiated by “tonsillitis”. The rules are all set to be “refracted” which is to say they will only be applied once with any specific instantiation, though they can be applied any number of times with different instantiations. Apart from the reduction in rule numbers from the original model there are a number of other changes to the form of the rules in the revised model. First, COGENT processes can manipulate any number of buffers. Each condition of a rule, therefore, refers explicitly to a buffer, here “working memory” or “knowlege base”. This means that COGENT will test the conditions of the rules by attempting to retrieve matching terms from the named buffer. Second, the rules include variables, such as Disease, Symptom and so on. The syntactic form of a rule’s premisses constrains the form of the terms they can be matched in the buffer. In the terminology of mathematical logic the first model is limited to propositional reasoning, while the latter has the greater power of the first-order predicate calculus. Figure 9: Execution of the first-order implementation of the model. The behaviour of the model is indistinguishable from the propositional implementation of the decision procedure. Dysphagia Vomiting Headache Earache Pyrexia No question Dysphagia D V Vomiting DV Headache D V Earache D V Pyrexia DV Diagnostic accuracy by comparison with the true presenting disease: Average number of questions before giving a diagnosis: D 2.3; D 80 %; V 92 %; overall 83% V 3.1; av = 2.7 Table 9: Summary analysis for the first-order model showing that its behaviour is effectively identical to the behaviour to the propositional model. Returning to our general theme we have predicted that even apparently major changes to the representation need not imply any change in the behaviour of the model, so long as we do not alter either the medical knowledge content or the memory retrieval parameters,. Figure 9 shows the state of working memory at the end of a diagnostic trial; it is identical to that shown in Figure 4, and indeed the dynamic processing of working memory by the first-order decision procedure is the same as for the propositional procedure for all simulated patients. This is reflected in the summary data in Table 6. 4. Discussion We have suggested that the information processing mechanisms which many cognitive scientists seek to understand must underpin decision making as well as other cognitive skills, and that an understanding of their role is likely to form part of a complete theory of human judgement. The 1980 study of decision making suggested that an understanding of memory retrieval mechanisms, symbolic reasoning, and the interaction between them could make a significant contribution to our understanding of human decision making under uncertainty. The additional simulation experiments reported here underline this point, particularly the demonstration that assumptions about memory access can be crucial in achieving accurate predictions about decision making behaviour. Decision making can be productively viewed as a process of applying symbolic knowledge, both specialised and general knowledge, to symbolic data. This contrasts with classical algebraic models which view judgement as a form of numerical calculation. On the contrary the present approach, in which medical knowledge is represented as qualitative inference rules, leads to a model which appears both powerful and intuitively appealing as an account of how decision makers form hypotheses, seek evidence, take decisions and so forth. The equivalence of the propositional and first-order representations is particularly striking. From a purely logical point of view the result is not surprising (any theorem in propositional logic is a theorem in first-order logic) but from a cognitive psychological point of view it raises some interesting issues. Hastie and Pennington (1995) remark that a reliable observation from several decades of research on mental representations is that people are “concrete thinkers”; we prefer taskspecific, concrete representations of the world and rarely rely on abstract representations. The decision procedure in the propositional model reported here, is implemented as a set of concrete, “special case” rules. The first-order decision procedure, however, abstracts a generalised decision making process from knowledge of specific diseases and symptoms. One possible conclusion from this is that the level of abstraction which people actually use may not be easy to identify empirically, at least in the present type of task, since the concrete and abstract process models can produce indistinguishable behaviour. (In fact it is possible to carry out a further abstraction step, in which strategic knowledge of how to take any kind of decision is abstracted from the relatively specific knowledge of how to take diagnosis decisions in medicine and still exactly simulate the behaviour of our subjects 6.) This raises the general issue of distinguishability of theories, which has been debated in a number of areas of cognitive psychology. John Anderson and others argue that representations cannot, in principle, be distinguished empirically, for example, since any representation of sufficient richness can in principle mimic any other, while Zenon Pylyshin 6 This is not reported here because the further abstraction step appears to require the introduction of a number of very general “mentalistic” concepts, such as goals and plans, which are intuitively plausible but for which we have no empirical justification. and Steven Kosslyn have argued that specific cognitive representation (propositional and analogical representations respectively) make distinct and empirically testable predictions. A question which arises from the indistinguishability of propositional and first-order knowledge representations concerns whether human knowledge is encoded declaratively or procedurally and whether the cognitive implementations of these two types of memory are different. The encoding and retrieval procedures for these knowledge types have been claimed to involve different mechanisms, which are manifested in differences in people’s ability to introspect on their knowledge, the occurrence of different retrieval errors, and so forth. If there are such differences they are not evident in the COGENT model. The distinction between procedural and declarative knowledge has found a role in understanding other kinds of cognitive skills, and even that different representations may be used at different stages in development and at different levels of expertise. For example Karmiloff-Smith (1992) has argued that development, and perhaps learning, involves a process of “representational redescription”, whereby an initial procedural representation is revised as learning progresses and is eventually replaced by a declarative representation as mastery in the domain is attained. Recalling some of the a priori advantages of the first-order representation this issue might have important theoretical implications for our understanding of human decision making, and practical consequences for training decision makers. On a somewhat related point Hastie and Pennington suggest that “special case” knowledge is more frequently acquired than abstract, generalised knowledge. We are inclined to agree with this, but any conclusion that this is a fundamental property of human cognition does not necessarily follow. Hastie and Pennington themselves accept that “deliberately trained expertise” may well include abstractions. In the light of the possible indistinguishability of the propositional and first-order models we need to be cautious about strong claims regarding whether decision makers naturally use, or eschew, abstractions. The empirical observation of a bias towards concrete representations may be correct, but that might be an artifact of the learning process or other aspects of experience. We tend to acquire specific knowledge in specific situations, of course, but often we simply do not have the time, inclination or sufficiently varied examples to discover any generalisations that may exist. This would be a much more modest conclusion than a claim that human decision processes are not naturally organised for exploiting abstractions. Finally we suggest that computational modelling tools like COGENT can make a significant contribution to the understanding of decision making, particularly complex, practical decision making. Psychological processes are subtle and difficult to understand, and the behaviour we observe is often the expression of many interacting processes. As a consequence theories are often based on relatively simple experimental manipulations, whose analysis is mathematically tractable, but the conclusions may not extrapolate to complex, real world tasks. Among our own objectives are the desire to develop a general theory of complex decision making, which can have practical value. The particular “real world” that we are primarily concerned with is medicine; the problems we wish to address include: how to design computer systems to help medical professionals in their routine decision making and patient management (e.g. Fox and Das, 1996) and how to communicate information about risks and uncertainties in ways that people can understand (e.g. Fox et al, 1995). Our experience to date suggests that these design objectives are considerably helped by insights into how people make decisions and not just what decisions they will take. 5. Conclusions An eclectic approach to theories of human judgement and decision making is needed, combining insights from the algebraic, representational and information processing perspectives. We agree with Busemeyer et al’s observation that “cognitive scientists generally find that decision theories fail to provide sufficient depth of explanation, and decision theorists generally find that cognitive theories are too complex”. We hope that the methods and results presented here provide some support for our belief that cognitive modelling in general, and tools like COGENT in particular, can help to ameliorate such problems, and bring the JDM communities closer together. 6. References Beach L R Image theory: decision making in personal and organizational contexts. Chichester, England: John Wiley, 1990. Busemeyer J, Hastie R and Medin D L (eds) Decision making from a cognitive perspective San Diego: Academic Press, 1995. Cooper R C and Shallice T, “Soar and the case for unified theories of cognition”, Cognition, 55(2), 115-149, 1995. Cooper R C, Fox J, Farringdon J and Shallice T “A systematic methodology for cognitive modelling” Artificial Intelligence Journal, August 1996 (in press) Fishburn P C “Normative theories of decision making under risk and under uncertainty” In Bell, Raife and Tversky (eds) Decision Making, Cambridge: Cambridge University Press,1988 Fox J “Making decisions under the influence of memory” Psychological Review, 1980 Fox J “Automating assistance for safety critical decisions” Phil. Trans. Roy. Soc. Lond. B, 327, 447-593, 1990. Fox J “On the necessity of probability: reasons to believe and grounds for doubt” in G Wright and P Ayton (eds) Subjective Probability, Chichester: John Wiley, 1994. Fox J, Clark D A, Glowinski A J and O’Neil M “Using predicate logic to integrate qualitative reasoning and classical decision theory” IEEE Trans. Systems, Man and Cybernetics, 20, 347-357, 1990. Fox J and Das S “A unified framework for hypothetical and practical reasoning (2): lessons from medical applications” in D M Gabbay and H J Ohlbach (eds) Practical Reasoning, Berlin: Springer Verlag, 1996. Fox J, Hardman D, Krause P, Ayton P, Judson, P “Risk assessment and communication: a a cognitive engineering approach” in A Macintosh and C Cooper (eds) Applications and innovations in expert systems III, SGES Publications: ISBN 1 899621 03 2, 1995 Gigerenzer G “Why the distinction between single-event probabilities and frequencies is important for psychology (and vice versa)” in G Wright and P Ayton (eds) Subjective Probability, Chichester: John Wiley, 1994. Hastie R and Pennington N “Cognitive approaches to judgment and decision making” in Busemeyer J, Hastie R and Medin D L (eds) Decision making from a cognitive perspective, San Diego: Academic Press, 1995. Karmiloff-Smith A “Beyond modularity: a developmental perspective on cognitive science” Cambridge, MA: MIT Press. Shanteau J “Psychological characteristics of expert decision makers” in J Mumpower (ed) Expert judgement and expert systems, NATO ASI Series, vol F35, 1987. Wallsten, T S (ed) Cognitive processes in choice and decision behavior, Hillsdale NJ: Erlbaum, 1980. Weber, E U, Goldstein W M and Barlas S “And let us not forget memory: the role of memory processes and techniques in the study of judgement and choice” in Busemeyer J, Hastie R and Medin D L (eds) Decision making from a cognitive perspective San Diego: Academic Press, 1995.