The Effect of Navigation Maps on Problem Solving
Tasks Instantiated in a Computer-Based Video Game
Committee Members: Dr. Harold O’Neil (Chair)
Dr. Edward Kazlauskas
Dr. Yanis Yortsos (Outside Member)
Richard Wainess
Dissertation Presented to the
Faculty of the Graduate School
University of Southern California
December 9, 2005
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 1 of 14
Research Hypotheses
Problem Solving
– Hypothesis 1: Participants who use a navigation map (the treatment group) will
exhibit significantly greater content understanding than participants who do not use
a navigation map (the control group).
– Hypothesis 2: Participants who use a navigation map (the treatment group) will
exhibit greater problem solving strategy retention than participants who do not use
a navigation map (the control group).
– Hypothesis 3: Participants who use a navigation map (the treatment group) will
exhibit greater problem solving strategy transfer than participants who do not use a
navigation map (the control group).
– Hypothesis 4: There will be no significant difference in self-regulation between the
navigation map group (the treatment group) and the control group. However, it is
expected that higher levels of self-regulation will be associated with better
performance.
Motivation
– Hypothesis 5: Participants who use a navigation map (the treatment group) will
exhibit a greater amount of continuing motivation, as indicated by continued
optional game play, than participants who do not use a navigation map (the control
group).
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 2 of 14
Cognitive Load Theory
• Cognitive Architecture
– Auditory/Verbal and Visual/Spatial Channels/Memory
(Baddeley,1986; Mayer & Moreno, 2003)
– Limited Working Memory (Brunken et al., 2003)
• 7 +/- 2 (Miller, 1956)
• Possibly only 2 or 3 novel elements (Paas et al., 2003)
– Unlimited Long-Term Memory (Mousavi et al., 1995)
• Cognitive Load = mental capacity imposed on working memory
(Sweller & Chandler, 1994)
– Controlled by schema development & Automation
– Intrinsic Cognitive Load: Necessary (Brunken et al., 2003; Paas et al., 2003).
– Germane Cognitive Load: Required to process intrinsic load (Renkl &
Atkinson, 2003).
– Extraneous Cognitive Load: Unnecessary stimuli (Brunken et al. 2003).
• Seductive Details (Mayer et al., 2001; Schraw, 1998).
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 3 of 14
Scaffolding
(including Graphical Scaffolding)
• Instructional methods
– Should keep cognitive load low (Clark, 2003)
– External methods which replace internal processes (Clark, 2001)
• Scaffolding is an instructional method
• Scaffolding provides support during learning
(Allen, 1997; Chalmers, 2003; van Merrienboer et al., 2002, 2003)
• Graphical Scaffolding
– Includes maps and menus as advance organizers (Jones et al., 1995)
– Maps supported by researchers as visual aids and organizers
(Benbasat & Todd, 1993; Chou & Lin, 1998; Ruddle et al., 1999)
– Should be used for visual tasks (Mayer et al., 2002)
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 4 of 14
Navigation Maps
• Navigation: Tracking one’s position in an environment to arrive
at a destination (Cutmore et al., 2000)
• Occlusion: when a path is blocked visually (Cutmore et al., 2000)
• Navigation maps effective for occluded 3-D navigation
(Cutmore et al. 2000; Dempsey, 2002)
• Navigation maps effective in 2-D environments with complex
problem solving tasks (Baylor, 2001; Chou & Lin, 1998; Chou et al., 2000)
• Navigation maps effective in 3-D occluded environments with
simple problem solving tasks (Galimberti, 2001)
Not yet examined
Navigation in 3D, occluded, environments
with complex problem solving tasks
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 5 of 14
Navigation Map
Floor Plan of Mansion’s First Floor from SafeCracker®
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 6 of 14
Method: Design
• Design
– Experimental, Posttest Only, 2x2 Repeated Measures
• First factor—two levels (one treatment group, one control group)
• Second factor—two levels (occasion 1, occasion 2)
– Random assignment to treatment group or control
group
– Treatment receives navigation map; Control Group
doesn’t
– Pilot study and main study
– Approx. 90 minutes
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 7 of 14
Method: Participants
• Participants (paid $15 for participation)
– Main Study (November 11, 2004 through March 21, 2005)
• 71 English-speaking males and females
– Data analyzed for 64 (33 treatment, 31 control)
• Undergraduate and graduate students at USC
• No prior experience with the game
SafeCracker®
• Average Age: a few days short of 29 years
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 8 of 14
Method: Timing Chart for Main Study
Introduction and study paperwork
Self-regulation and demographic questionnaires
Introduction to knowledge mapping software
Introduction to SafeCracker
Introduction to map reading for the treatment group
FIRST GAME (3 rooms) plus task completion form
Knowledge map creation (occasion 1)
Problem solving strategy retention and transfer questionnaire (occasion 1)
SECOND GAME (3 rooms) plus task completion form
Knowledge map creation (occasion 2)
Problem solving strategy retention and transfer questionnaire (occasion 2)
Debriefing
Optional additional playing time
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 9 of 14
Results
Problem Solving
– Hypothesis 1: Participants who use a navigation map (the treatment
group) will exhibit significantly greater content understanding than
participants who do not use a navigation map (the control group).
NOT SUPPORTED
– Hypothesis 2: Participants who use a navigation map (the treatment
group) will exhibit greater problem solving strategy retention than
participants who do not use a navigation map (the control group).
NOT SUPPORTED
– Hypothesis 3: Participants who use a navigation map (the treatment
group) will exhibit greater problem solving strategy transfer than
participants who do not use a navigation map (the control group).
NOT SUPPORTED
Motivation
– Hypothesis 5: Participants who use a navigation map (the treatment
group) will exhibit a greater amount of continuing motivation, as indicated
by continued optional game play, than participants who do not use a
navigation map (the control group).
NOT SUPPORTED
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 10 of 14
Results (cont’d)
Problem Solving
– Hypothesis 4: There will be no significant difference in self-regulation
between the navigation map group (the treatment group) and the control
group. However, it is expected that higher levels of self-regulation will be
associated with better performance.
– PARTIALLY SUPPORTED
• Navigation Map Group: Positive correlation between mental effort
and problem solving retention improvement
• Control Group: Negative correlation between planning and problem
solving transfer improvement
• Self-efficacy not supported
• Monitoring not supported
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 11 of 14
Discussion
• Reduction of performance for treatment group
offsetting effect of navigation map
– Contiguity Effect (Mayer et al., 1999; Mayer & Moreno, 2003; Mayer & Sims,
1994; Moreno & Mayer, 1999)
• Spatial and temporal
• Results in Split Attention Effect (Atkinson et al., 2000; Mayer, 2001;
Tarmizi & Sweller, 1998)
– Cognitive strain during integration of information
– Extraneous Load (Harp & Mayer, 1998; Mayer, Heiser, & Lonn, 2001; Moreno &
Mayer, 2000; Renkl & Atkinson, 2003; Schraw, 1998)
• Seductive Details (Mayer et al., 2001; Schraw, 1998)
– Affects retention and transfer (Moreno & Mayer, 2000)
• Navigation Map?
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 12 of 14
Discussion (cont’d)
• Increase in performance for both groups, offsetting
effect of navigation map
– Strategy Priming
• Ready mind to allow or engage particular relevant schema
(http://filebox.vt.edu/8080/users/dereese2/module8/module08bkup/IDProjectWebpage/lesson4.htm)
• Repetition closely aligned with skill acquisition (Dennis & Schmidt,
2003)
– Lack of priming reduces process of selecting (Moreno & Mayer, 2005)
• Primed during:
– Knowledge map training and SafeCracker training
– During navigation map and basic navigation training
– Start of each game
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 13 of 14
Implications
• Navigation maps do not guarantee improved
performance
• Further studies are suggested
– Manipulation of contiguity effect
• Integrated versus separated map
– Manipulation of extraneous details
• Low versus high levels of environmental fidelity (both with map use)
• Small versus large environment (both with map use)
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 14 of 14
BACK UP
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 15
Problem Solving Assessment
• Based on O’Neil’s (1999) Problem Solving Model
– Requires content understanding, problem solving strategies, selfregulation
• Transfer questions are alternative to transfer tasks (Moreno & Mayer, 1998)
• Declarative knowledge measured by retention (Day et al., 2001)
• Knowledge map: concepts and links (Schau & Mattern, 1997)
– Reflects the organization of knowledge (Day et al., 2001)
– Reliable and efficient measure of Content Understanding
(Herl et al., 1999; O’Neil, 1999; Ruiz-Primo et al., 1997; Schacter et al., 1999)
Predictive of both retention and transfer (Day et al., 2001)
–
– Reliable measure of Problem Solving Strategies (Baker & Mayer, 1999)
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 16
Problem Solving Assessment (cont’d)
• Measuring Problem Solving Strategies
– Domain general and specific (Alexander, 1992; Bruning et al., 1999)
– Knowledge Mapping (Baker & Mayer, 1999)
– Problem solving questions
• Positively correlated with retention and transfer (Mayer & Baker, 1998)
• Measuring Self-Regulation
– Includes Metacognition & Self-efficacy (O’Neil, 1999)
• Metacognition: planning & self-checking (Pintrich & DeGroot, 1990)
• Self-efficacy: mental effort and self-efficacy (Zimmerman, 1994, 2000)
– Trait self-regulation questionnaire (Hong & O’Neil, 2001)
• Includes planning, self-checking, mental effort, and self-efficacy
• 32 questions: 8 for each of the four sub-categories
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 17
Games and Simulations
• Games: rules, constraints/privileges, imaginative, linear (Gredler, 1996)
• Simulation-Games: combination of games and simulations (Gredler, 1996)
• Motivation in games: fantasy, control & manipulation, challenge & complexity,
curiosity, competition, feedback, fun
• Positive outcomes:
– Numerous knowledge outcomes attributed to games and simulations
• Warning about anecdotal and descriptive evaluations (Leemkuil et al., 2003; Wolfe, 1997)
– Generalizable skills outcomes
(Day et al. 2001; Green & Bavelier, 2003; Greenfield et al., 1994)
• Negative or null outcomes:
– Reviews and meta-analyses cite mixed or negative reviews
(Dekkers & Donatti, 1981; Druckman, 1995)
– Positive attitude toward games doesn’t necessarily equal learning
(Brougere, 1999; Salas et al., 1998; Salomon, 1984)
• Outcomes related to Instructional Design, not games/simulations
(de Jong & van Joolingen, 1998; Garris et al., 2002; Gredler, 1996; Leemkuil et al., 2003; Thiagarajan, 1998)
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 18
O’Neil’s Problem Solving Model
Problem Solving
Content
Understanding
Problem solving
Strategies
Self-Regulation
Metacognition
Planning
Domain
Specific
SelfMonitoring
Motivation
Effort
Domain
Independent
O’Neil (1999)
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 19
Content Understanding:
Knowledge Mapping Software
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 20
Sample SafeCracker® Knowledge Map
desk
brochure
contains
contains
contains
results from
contains
room
contains
key
part of
clue
causes
safe
contains results from
results from
contains
causes
part of
crack
uses
books
used for
tool
Richard Wainess Dissertation v.4
used for
causes
direction
used for
Presented 12/09/05
map
Slide 21
Problem Solving Strategy
Retention and Transfer Questionnaire
Retention Question:
1. List the ways you found rooms and opened safes
Transfer Question:
1. List some ways to improve the design of the game
play for opening safes
• 28 idea units generated for Retention Question
• 21 idea units generated for Transfer Question
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 22
Self-Regulation Questionnaire
• Based on O’Neil (1999) Problem Solving Model
• Trait self-regulation questionnaire (O’Neil & Herl, 1998).
• 32 Questions: 8 each of four measures
–
–
–
–
planning
self-checking/monitoring
self-efficacy
effort
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 23
Data Analysis
• Descriptive statistics: Means, Standard Deviation, etc. for
content understanding, problem solving strategy retention and
transfer, trait self-regulation, number of safes opened, and
continuing motivation.
• Mixed Group, Repeated Measures Factorial ANOVA:
Examine effect of use/non-use of navigation map on content
understanding, problem solving strategy retention and transfer,
and number of safes opened.
• Correlations: between self-regulation components (planning,
monitoring, effort, self-efficacy) and content understanding,
problem solving strategy retention and transfer, and number of
safes opened for each group and both groups combined.
Richard Wainess Dissertation v.4
Presented 12/09/05
Slide 24