Version 0.5 25. November 2003 Review of the research on educational usage of games Simon Egenfeldt-Nielsen IT-University Copenhagen Glentevej 67 DK-2400 NV Copenhagen +45 40 10 79 69 egenfeldt@game-research.com Abstract The goal of this paper is to make it possible to overview the area of educational games before embarking on new research. The paper examines the current research evidence for and against the usage of games for educational purposes. The paper will also outline two important areas that have so far been identified as important for the effectiveness of games: Learning environment and personal learning factors. It is concluded that these areas are very important areas but often they are not well documented nor researched to a large extent. In relation to the effect of using games it is concluded that there is currently no evidence for a better or worse learning outcome, when games are used. Although there is indication that retention might be better when using games for learning. Keywords Games, computer games, education, learning, review, simulation, effectiveness Version 0.5 25. November 2003 Introduction There should be no doubt in our mind that we can learn something from games just like other activities we engage in. The discussion is rather what we learn and how effective the learning process is. Following this discussion another question often arises. Namely whether games have a place in educational settings, and if so under what conditions in relation to for example subject, setting, and age group. Although this question is not new it remains central to the study of simulation and games for learning (see for example Boocock & Schild, 1968). Although this paper focuses on non-electronic games it is my hope that this article can be used to transfer some of the research from educational games to the younger research field educational computer games. There is good reason to believe that we face similar problems in the two fields (Egenfeldt-Nielsen, 2004). In the following I will not distinguish between simulations and games on an overall plan, as I believe the majority of simulations that have been used over the years can to some degree be described as games. Hereby I follow the distinction by Egenfeldt-Nielsen & Smith (2000) where simulation and games is separated by the existence of goals. Simulation games do not have explicit goals as such, whereas these are central to games. Furthermore it seems that setting up goals can transfer any simulation into a game. This is a general definition which is useful for this article although it somewhat simplifies the distinction between simulations and games. When we combine the research areas learning, education, games, and simulation we have a very cross-disciplinary discipline, which inherit several tough and controversial research questions from their respective areas. One of the main problems of presenting the results below is that the evaluation methods in relation to games is a problem in themselves, where it has been questioned if we can use traditional methods for measuring learning outcome (Druckman, 1995). I will try to address some of these concerns towards the end of the paper. Some researchers claim that we simply learn to play the game in question while other subtler argues for a transfer of knowledge, where the elements we experience within in the game context can be used in other areas of our life. This question is interesting and relevant but is not within the scope of this paper. In this paper I assume that we are actually in some way capable of transferring knowledge between different contexts. In summary this paper will survey the research that has so far been carried out into the effectiveness of games. The focus is on providing an overview rather than going in detail with specific studies. Version 0.5 25. November 2003 A frame for reviewing learning through games The following examination of different important aspects of games promoting learning is based on the categories in earlier review studies. It has not been possible to go into detail with separate studies but instead I aim to present an overview of the most important review articles over the years (Wentworth & Lewis, 1973; Bredemeier & Greenblat, 1981; Van Sickle, 1986; Dorn, 1989; Clegg, 1991; Randel et al., 1992; Dempsey et.al, 1996). It should be noted that the reviews are quite dated, which is a sign of the decline of research in non-electronic educational games in favour of research on educational potential of computer games (Egenfeldt-Nielsen, 2003). In some places I will use articles of newer date to present areas not covered sufficiently in existing reviews. The review has been split into three areas. Each of these areas will be presented and discussed in the paper to give a status of the research field. 1. Learning environment: What properties of a learning environment have a bearing on the learning outcome, and what variables are important. 2. Personal learning factors: What personal factors can play a role for the learning outcome of a specific game experience, and how significant is the impact. 3. Learning outcome: What are the effects of using games for learning within different areas and in relation to different domains of learning. Before entering into the specifics of these areas it is worth considering the more overall factors like the type of games used, and within what subjects. It seems obvious that games could lend themselves better to some subjects, and that not all game types have been equally used. Early on there were few been studies comparing the advantages and differences between different game genres (Roberts, 1976) but from the literature it is clear that simulation type is preferred. Research that compares the suitability of different types of games for learning is still limited. In a search on what game types were encountered in articles on simulation and gaming Dempsey et al. (1996) found 43 simulation games, 26 other, 10 adventure, 4 puzzles, and 1 experimental. They attribute the large number of simulations to the history of the research, which grows out of business1 and military training (Roberts, 1973). The problem with their distinction is that the simulation category is quite broad, as simulation is in a majority of games an almost intrinsic property. 1 Especially business education have taken games to their heart perhaps best exemplified with the following web-site that list 140 games specifically developed for business http://www.marietta.edu/~delemeeg/games/ Version 0.5 25. November 2003 The learning environment The role of the learning environment has been studied from different angles focusing on the significance of subject area, education level, motivation, the use of debriefing, administration, and instructor type. It seems that games have from the start especially been used within the areas social sciences, math, business, and military (Dukes & Seidner, 1978; Butler, Markulis & Strang, 1988; Prensky, 2001), which are also the areas where the research efforts has been greatest. Randel et al. (1992) finds that social studies top the list with 46 conducted studies, 8 studies within math, 6 in language arts, 3 in biology, and 1 in physics. Over the years the use of games has spread to all areas of the education system but with primary and secondary education lacking behind compared to tertiary education (Ellington, 1995). This corresponds with Dempsey et al. (1996) that find that the use of games in adult learning is the most researched area. The success of games in tertiary education might be explained as a consequence of, what is socially acceptable. Research into adults using games for learning is probably perceived as more serious as adults don’t play for fun. This is serious research of a serious tool for learning. Furthermore the teaching of adults with games might be conceived as more serious as adults according to Sutton-Smith & Kelly-Byrne (1984) are not expected to play. Therefore games are constructed as a serious activity for adults when in an educational setting. For children on the other hand teaching with games is perceived as nothing else than play - and do they really learn anything from that? The difference cannot be explained due to better effect in adult learning as Dempsey et.al. (1996) do not find any indication that games are more effective for adult teaching. The importance of motivation and interest It has be claimed that the overall atmosphere of the learning environment improve when games are used, a claim which Seidner (1978), Bredemeier & Greenblat (1981), and Dorn (1989) support, however the evidence is not convincing. Still the idea of an improvement in the overall atmosphere ties well into one of the most significant variables concerning the learning environment of games: The increase in students’ motivation and interest. This factor is identified as far back as 1968 and continues to be central. The majority of researchers like Boocock, & Schild (1968), Wentworth & Lewis (1973), Coleman et al. (1973), Seidner (1978), and Bredemeier & Greenblat (1981). Dorn (1989), Clegg (1991) and Randel et al. (1992), support this claim. However Version 0.5 25. November 2003 despite the common agreement of games increasing motivation and interest, it still remains to be seen, what the specific implications are of this increase and whether the increased motivation relates to a better learning experience. Dorn (1989) questions whether the increased motivation for specifically games holds true when you compare games with more active teaching methods like case studies. Debriefing – an indispensable tool Debriefing is another classic issue, which from the start has been perceived as an important element in the learning environment although the early Hopkins Games Program (Coleman et al, 1973) did not find that a class discussion after the game increased the learning outcome, however this may be due to a weak guided debriefing process or limited capability of the teacher that conducted the debriefing (Seidner, 1978). Later research support debriefing for example Bredemeier & Greenblat (1981:310), which state that debriefing is “essential for maximum (or even correct) learning to occur”. Linda Costigan Lederman (1984:426-429) follows up on the importance of debriefing for correct learning stressing the need for the teacher to validate the learning experience, and check how reliable the outcome is. Furthermore she emphasize the need to assess the exercise from a cost-benefit perspective, an element that is still highly relevant, as we need to compare the learning outcome with the extra time spend on for example learning the game interface, setting up the game, and additional time for teachers preparation. The debriefing is a way to increase reflection over games that can sometimes be quite hectic (Thatcher, 1990). According to Bredemeier & Greenblat (1981) and Lederman & Fumitoshi (1995) the importance of debriefing is well recognized but the research is limited. In a recent article Charles Petranek (2000) backs up this claim and points to the lack of written debriefing. He argues that written debriefing is the next natural step in a learning process, and it is surprising that research into written debriefing is almost non-existing. The scourge of the setting The setting of games is usually thought of as the school, and this holds certain problems related to the quite rigid structure of the school system. Francois Saegesser (1981; 1984) and Coleman et al. (1973) points out that the time schedule, physical space, learning theory, and existing role scripts hinders the use of games, and this potentially influence the learning outcome. The influence of these factors are hard to access, however the different roles administration and instructors play, should alert us to the implications of the overall structure in the school system. Bredemeier & Greenblat Version 0.5 25. November 2003 (1981) and Dorn (1989) state that the attitude of the instructor toward games influences the outcome, and that the instructor’s knowledge and skill in using the game is also a factor. Furthermore the concrete use of the game can change the outcome if the instructor uses a variation of the gameplay or introduces the game differently. The importance of the learning setting has been somewhat neglected in the research over the years perhaps as a consequences of the more rich hunting fields in adult learning especially business and military training. Here the constraints and traditions of learning are less tainted by time. The instructors role Some demands can be made on the instructor for achieving the best learning outcome. Dorn (1989) outline some of them, and in accordance with this review I have added some. The question of how to use games have not been discussed in this paper but it worth noting that according to Dorn (1989) games should not be the only teaching method in a course, and the game sessions should be supported by other teaching methods and have a clear aim. Depending on the use and timing of games in a course, they can serve different purposes for example games in the start of a course can serve to introduce theory and concept while giving the students a way to know each other. More specifically the instructor should have some experience with the games used, be skilled in administering the game, have a desire to use the game, keep in mind the role scripts in the class, be able to perform debriefing, evaluate different games, and be aware of the physical limits with regards to time and space in school. These seem like basic and reasonable demands, however they may prove much harder to implement in real life. These recommendations are also in line with Charles Elders (1973) findings. The complexity that opens up with the many variables in the learning environment are staggering and quite overwhelming for anyone with a practical focus although some seem convinced these can be overcome (Dorn, 1989). Personal learning factors The role of personal characteristics has been examined to some degree although Dempsey et al. (1996:12) finds that in reality studies are “very unclear in reporting these characteristics”. In the review the most frequently reported variables were gender, age, academic ability, and to some degree race. This is quite interesting as the early work in the field stress the importance of individual differences (Greenblat, 1981), and the importance of examining the sample to avoid method problems related to matched Version 0.5 25. November 2003 samples (Remus, 1981), which is often a given experimental design as you must conduct the study within the existing class structure in schools. The fact that basic factors like age, race, gender, and academic ability is only cited in approximately 25% of the studies indicate the problems are not yet resolved. One area that has consistently received attention is whether games favour the students with the best academic abilities or is perhaps a mean to reach the students with weaker academic abilities. According to Coleman et al. (1973) the games are more effective with students with high academic ability, however Bredemeier & Greenblat (1981) finds more mixed results and points out that games may be able to reach less advantaged student groups but this depends on other factors as well. They especially point out that the prior attitude of the student towards the game plays a role. Other indications of the role of personal factors are the structure of the group in relation to size, organization, and relations between students (Clegg, 1991). The argument is that the composition of the group facilitates different learning experiences and dynamics in both the environment and for the specific student for example winners and losers of the game have very different experiences (Bredemeier & Greenblat, 1981). It seems that the initial group cohesion is quite important for the performance in a game environment, and remains relatively stable as the game progress (Wellington & Faria, 1996). In the study by Wellington & Faria (1996) the initial attitude of the students is found to be of limited consequence for the performance, whereas group cohesion is central. One area that has received little attention is the importance of earlier experience with games, which could be expected to play a role for learning outcome. Knowledge and experience with playing games would make it possible for the student to concentrate on playing the game, and spending less time mastering a new learning form. Learning outcome This is one of the areas, which have received the most attention over the years; how can we measure the learning effect of games on users. As I have implied earlier this question is way to broad, and needs to be qualified with: “For what purpose, under what conditions, and how can we be sure.” (Bredemeier & Greenblat, 1981:307). This is what I have tried to clear the way for in the sections above, however I will still try to give a general verdict on the learning outcome associated with games in the following. The claims for what games can do is very long even in 1981 when Greenblat (1981) sums it up. Greenblat’s list is too long to reproduce here but in short forms it spans two pages with 30 different claims. Version 0.5 25. November 2003 A Procrustean bed: Assessing learning outcome The 30 claims and other claims over the years 2 have been hard to document due to problems with measuring learning in a school context and to construct methods for measuring in accordance with existing dominating learning theories. Also it has been pointed out that several studies in general have method flaws (Dorn, 1989). These problems have been clear from early on but continue to influence the research. Bredemeier & Greenblat (1981) and Dorn (1989) points out that we should be aware of what basic learning theory we ascribe to, and what assessment is feasible to confirm learning outcome within our theoretical frame. Saegesser (1981) points out that existing learning theories in schools put external signs of learning over more subtle forms. Although these notions of warning are more than 20 years old, they still hold true although the theoretical landscape of learning have changed significantly (Gardner, 1983). We still run the risk of not realising the full potential of games if we try to put them into a Procrustean bed. On the other hand we need to be specific about the learning outcome we expect, and find other ways that we can measure learning. Otherwise we run the risk of the learning being repetitive, undocumented, confusing and pointing in different directions, which is a very real risk, when using games for teaching (Elder, 1973). The assessment of learning has given rise to numerous problems over the years. One of the most persistent is the lack of taking into consideration the most basic factors when conducting a study. According to Joseph Wolfe and David Crookall (1998) these problems are partly related to the research area being quite eclectic. This has led too less rigorous standards for assessment. In line with Elder (1973), Bredemeier & Greenblat (1981) state that three areas need to be measured to gauge learning outcome in a given game. The overall variable is substantive learning, which can be both learning about a subject or about oneself (cognitive and affective). When assessment of learning outcome is divided into cognitive and affective we can distinguish between different domains of knowledge and learning methods. The split between cognitive and affective is also sometimes seen as the division of specific skills and more general principles, concepts, and orientations. The problems with distinguishing between these two areas have influenced game research from the start and the problem is shared with other educational research (Elder, 1973). The two other areas that as a minimum should be assessed are those having a bearing on the overall variable learning outcome. These are motivation to learn and the relation 2 For example the loose claims at the Educational Simulations web-site http://www.creativeteachingsite.com/edusims.html Version 0.5 25. November 2003 between students and teacher. Most of these variables are taken into consideration in serious new studies for example the study of Pomp and Circumstances by Meguni Kashibuchi and Akira Sakamoto (2001). The outcome of cognitive and affective learning The reviews are not consistent in their conclusions on learning outcome but in general it can be concluded that: In some instances games seem to be more appropriate for affective learning, while cognitive learning in general is found to be as good as other teaching methods (excluding retention over time) (Bredemeier & Greenblat, 1981; Dorn, 1989; Van Sickle, 1986, Randel et.al, 1992). The review by Dempsey et al. (1996) following Gagne’s taxonomy3 shows that problem solving is the most researched learning outcome followed by attitudes. This imply that in general much of the research have been aimed at identifying learning outcome in more overall competences, and not necessarily the learning outcome of facts and concepts. This also ties in well with the overweight of research in social sciences, where problem solving and attitudes have traditionally been a focus (Abt, 1968:70-71). The success of math games reported by Randel et al. (1992) to some degree runs counter to this, as math is a subject that learns very specific skills. Randel et.al. (1992) review concludes that the most successful learning outcome is achieved in games with very specific required skills. This implies that it may be relevant to revisit games for learning more specific skills for example in algebra, physics, chemistry or grammar rules. The increased retention over time of learning appears to be one of the most consistent finding within the research into the potential of games for learning (Bredemeier & Greenblat, 1981; Randel et al., 1992; Van Sickle, 1986), however Dorn (1989) challenges this claim citing older studies that show no difference in retention over time. However the newer and more thorough study by Randel et al. (1992) supports the claim for better retention over time. Although not all the studies in the meta-analysis by Randel et al. (1992) examine retention over time the ones that did, find a better retention over time compared to traditional teaching methods. 3 Attitude, motor skills, cognitive strategy, problem solving, rules, defined concepts, verbal information, other. Version 0.5 25. November 2003 Conclusion We have seen that games are used in very different subjects with varying degree of success. Some of the popular areas for games have been business, math, social studies, science, history, geography, and military. You may end up concluding that games have been used in almost any imaginable setting, subject, and age group, which is not entirely wrong. Within all these areas games are used for different purposes and it is still unclear what exactly defines games. We have indications that this may be a worthwhile area to study but we still need to become much more specific in terms of what games we are talking about. Although research into the specifics of each game and game types is limited, I firmly believe that a lot of the contradictory results stem from too little attention to the game in question. It does matter what kind of game we are studying, and more importantly we can have a large number of badly designed games. We should be careful not to equate a specific bad or extraordinary game with a general claim for using games for learning. A lot of the early studies on learning and games used games that were developed by fiery souls and with little training in designing games. It was learning by doing and some of the results from these first educational games are still with us today. These results may not show problems with games per se but rather with badly designed games. There are still a lot of problems left, when we look at learning and games but it would be a shame if we didn’t build on the rich results from the early days. The area stands a good chance of revitalization if it lends itself more to computer game research that is growing explosively these years. References Abt, Clark (1968). Games for Learning. In: Boocock, Sarane S. & Schild, E.O. (1968)(eds.). Simulation Games in Learning. London: Sage Publications. Boocock, Sarane S. & Schild, E.O. (1968)(eds.). Simulation Games in Learning. London: Sage Publications. Bredemeier, M.E., & Greenblat, C.S. (1981). The educational effectiveness of simulation games: A synthesis of findings. Simulation & Games, Vol. 12, No. 3: 307-331. Butler, Markulis & Strang (1988). Where are we? An Analysis of the Methods and Focus of The Research on Simulation Gaming. Simulation & Games, Vol. 19, No. 1: 3-26. Cathy Greenblat & Richard E Duke (1981)(eds.). Gaming-Simulation: Rationale Applications. London: Sage Publications. Version 0.5 25. November 2003 Clegg, A.A. (1991). Games and simulations in social studies education. In: Shaver, J. P. (1991)(eds.). Handbook of research on social studies teaching and learning. New York: Macmillan. Coleman, J. S., et al. 1973. The Hopkins' Games Program: Conclusions from Seven Years of Research. Educational Researcher 2:3-7. Dempsey, John V., Rasmussen, Karen & Lucassen, Barbara (1996). The Instructional Gaming Literature: Implications and 99 Sources. University of South Alabama. Technical Report 96-1. Dorn, D. S. 1989. Simulation Games: One More Tool On the Pedagogical Shelf. Teaching Sociology Vol. 17:1-18. Druckman, Daniel (1995). The Educational Effectiveness. In: Crookall, David & Arai, Kiyoshi (1995)(eds.). Simulation and gaming across disciplines and cultures. London: Sage Publications. Duke, Richard E. & Seidner, Constance J. (1975)(eds.). Learning with simulations and games. London: Sage Publications. Egenfeldt-Nielsen, Simon & Smith, Jonas H. (2000). Den digitale leg – om børn og computerspil [Digital Play – About children and computer games]. Copenhagen: Hans Reitzels Forlag. Egenfeldt-Nielsen, Simon (2003). The current status of the research communities in games and computer games. Work in Progress. Egenfeldt-Nielsen, Simon (2004). Practical barriers in using educational computer games. On the Horizon. Under preparation. Elder, Charles (1973). Problems in the Structure and Use of Educational Simulation. Sociology of Education, Vol. 46: 335-354. Ellington, Henry (1995). The Future of Simulation/Gaming in Britain. In: Crookall, David & Arai, Kiyoshi (1995)(eds.). Simulation and gaming across disciplines and cultures. London: Sage Publications. Gardner, Howard. (1983) Frames of mind: the theory of multiple intelligences. New York: Basic Books. Greenblat (1981). Teaching with Simulation Games: A review of Claims and Evidence. In: Duke, Richard E. & Greenblat, Cathy (1981)(eds.). Principles of Practice of Gaming-Simulation. London: Sage Publications. Kashibuchi, Meguni & Sakamoto, Akira (2001). The educational effectiveness of a simulation/game in sex education. Simulation & Gaming, Vol. 32, No. 3: 331343. Lederman, Linda C. & Fumitoshi, Kato (1995). Debriefing the Debriefing Process: A new look. In: Crookall, David & Arai, Kiyoshi (1995)(eds.). Simulation and gaming across disciplines and cultures. London: Sage Publications. Lederman, Linda C. (1984). Debriefing: A Critical Reexamination of the Post experience Analytic Process with Implications for Its Effective Use. Simulation & Games, Vol. 15, No. 4:415-431. Leemkuil, Henny, Jong, Ton de & Ootes, Susanne (2000). Review of educational use of games and simulations. Twente: University of Twente. Version 0.5 25. November 2003 Loftus, Geoffrey & Loftus, Elizabeth (1983). Mind at Play: The Psychology of Video Games. New York; Basic Books. Prensky, Marc (2001). Digital Game-Based Learning. New York: McGraw-Hill Randel, J. M., B. A. Morris, C. D. Wetzel, and B. V. Whitehill (1992). The Effectiveness of Games for Educational Purposes: A Review of Recent Research. Simulation & Gaming Vol. 23, No. 3:261-276 Remus, William E. (1981). Experimental Design for Analyzing Data on Games – Or, Even the Best Statistical Methods Do Not Replace Good Experimental Control. Simulation & Games, Vol. 12, No. 1: 3-14. Roberts, Nancy (1976). Simulation Gaming: A Critical Review. US Department of Health & Welfare. National Institute of Education. Saegesser, Francois (1981). Simulation-Gaming in the Classroom. Some Obstacles and Advantages. Simulation & Games, Vol. 12, No. 3: 281-294. Saegesser, Francois (1984). The Introduction of Play in Schools: A Philosophical Analysis of the Problems. Simulation & Games, Vol. 15, No. 1: 75-96. Seidner, Constance, J. (1978). Teaching with Simulations and Games. In: Duke, Richard E. & Seidner, Constance J. (1975)(eds.). Learning with simulations and games. London: Sage Publications. Sutton-Smith, Brian & Kelly-Byrne, Diana (1984). The idealization of play. In Smith, Peter K. (ed) Play in Animals and Humans. Basil Blackwell, Inc; Oxford, England, 1984. Van Sickle, Ronald (1986). A Quantitative Review of Research on Instructional Simulation Gaming: A Twenty-Year Perspective. Theory and Research in Social Education, Vol. 14, No. 3: 245-264. Wellington, William J. & Faria, A.J. (1996). Team Cohesion, Player Attitude, and Performance Expectations in Simulations. Simulation & Gaming, Vol. 27, No 1:23-40. Wentworth, D.R. & Lewis, D.R. (1973). A review of research on instructional games and simulations in social studies education. Social Education: 432-440. Wolfe, Joseph & Crookall, David (1998). Developing a Scientific Knowledge of Simulation/Gaming. Simulation & Gaming, Col. 29, No. 1: 7-19.