Problem Solving Assessment in Games and Simulation Environments
Richard E. Mayer
University of California, Santa Barbara
Collaborators
Eva Baker
Roxana Moreno
Gayle Dow
Harold F. O’Neil, Jr.
Jennifer Dyck
William Prothero
Anne Fay
Jill Quilici
James Lester
Valerie Sims
Patricia Mautone
Hiller Spires
Sarah Mayer
William Vilberg
Supported by
National Science Foundation, Andrew Mellon Foundation
Problem Solving Assessment in Games and Simulation Environments
Introduction
What Is the Rationale for Games and Simulations?
What Are Instructional Games and Simulations?
What Is Problem-Solving Assessment in Games and Simulations?
Example 1: Assessment of Problem Solving in Computer Programming
Example 2: Assessment of Problem Solving in Playing Educational Games
Example 3: Assessment of Problem Solving in the Bunny Game
Example 4: Assessment of Problem Solving in the Design-A-Plant Game
Example 5: Assessment of Problem Solving in the Profile Game
Example 6: Assessment of Problem Solving in the Dr. Phyz Simulation
Example 7: Assessment of Problem Solving in an Aircraft Simulation
Conclusion
What Is the Rationale for Games and Simulations?
Bransford, Brown, & Cocking (1999)
Visual modes of presentation: “technologies can help people
visualize difficult-to-understand concepts…[so] students are
able to work with visualization…software”
Interactive modes of presentation: “new technologies are interactive
[so] it is easier to create environments in which students learn
by doing”
What Are Instructional Games and Simulations?
An instructional simulation:
1. Is a multimedia environment that models a to-be-learned
system.
2. Allows for user interaction by responding in rule-based ways to
user input.
3. Is intended to promote learner understanding of the to-belearned system.
An instructional game:
4. Also involves one or more players competing to achieve some
goal which is accomplished through understanding the subject
matter.
What Is Problem-Solving Assessment in Games and Simulations?
Types of Knowledge
Name
Facts
Definition
basic elements
Example
magnets are made of metal
Concepts
models or principles
how an electric motor works
Procedures
step-by-step processes how to add numbers
Strategies
general methods
how to break a problem into parts
Criteria for Problem-Solving Assessments
Interactivity criterion: involves interaction between the learner and
the computer-based system
Multimedia criterion: involves verbal and non-verbal forms of
representation (such as graphics)
Problem-solving criterion: involves solving problems
Seven Games and Simulations for Assessing the Learner’s Knowledge
1.
2.
3.
4.
5.
6.
7.
Computer programming
Educational games
Bunny game
Design-a-plant game
Profile game
Dr. Phyz simulation
Aircraft simulation
Programming in Logo
RT 90
FD 5
Purpose: To determine the cognitive consequences of learning Logo or
BASIC programming
Test
Knowledge
Description
Score(s)
Map test
Strategic
Given a neighborhood map and list
of locations draw a path connecting
them.
Given a neighborhood map showing
a path connecting various locations,
give verbal directions.
Number correct
Number correct
Command
prediction
Conceptual
Given a Logo command, draw
what happens when it is executed.
Number correct,
misconception type
Problem
Conceptual
Given a word problem, select a
Number correct
translation
Following
procedures
corresponding equation.
Strategic
Given a list of verbal commands,
describe the final output.
From: Fay & Mayer (1987), Mayer & Fay (1987), Mayer, Dyck, & Vilberg (1986)
Number correct
Command prediction problem
RT 90
Problem translation
A car rental service charges 20 dollars a day and 15 cents a mile to
rent a car. Find the expression for total cost C, in dollars, of
renting a car for D days to travel M miles.
a. C = 20D + 0.15M
b. C = 15D + 0.20M
c. C = 20D + 15 M
d. C = 0.15D + 20M
Following procedures
1. Put 5 in Box A.
2. Put 4 in Box B
3. Add the number in Box A and the number in Box B, put the
result in Box C.
4. Add the number in Box A and the number in Box C, put the
result in Box A.
5. Write down the numbers from Box A, B, and C.
What is the output of this program?
a. 5, 4, 9
b. 14, 4, 9
c. 14, 9, 9
d. 9, 4, 9
e. none of the above
Purpose: To determine the cognitive consequences of playing educational computer games.
Test
Knowledge
Description
Sentence
translation
Conceptual
Given a sentence from a word Number correct
problem, select the
corresponding number sentence.
Necessary
numbers
Conceptual
Given a word problem, select
the numbers that are needed to
solve the problem.
Necessary
operations
Strategic
Given a word problem, select
Number correct
the arithmetic operations that
are needed to solve the problem.
Puzzle
tank
Strategic
Given a new game, make
each necessary move.
Number of excess
moves, type of strategy
Mental
rotation
Strategic
Judge whether two shapes
are same or different.
Response time,
best fitting model
Form
board
Strategic
Judge whether shapes
fit together.
Response time
From Mayer et al. (1997), Mayer et al. (1999), Sims & Mayer (2002)
Score(s)
Number correct
Sentence translation
Ann and Rose have 20 books altogether.
Which number sentence is correct?
a. Ann’s books = Rose’s books + 20
b. Ann’s books + 20 = Rose’s books
c. Ann’s books + Rose’s books = 20
d. Ann’s books = Rose’s books
Necessary numbers
Which numbers are needed to solve this problem?
A package of 3 toys costs 88 cents. Richie bought 2 packages.
How many toys did he buy?
a.
3, 88, 2
b.
3, 88
c.
88, 2
d.
3, 2
Necessary operations
What should you do for this problem?
The 200 children at River View School are going on a bus trip.
Each bus holds 50 children. How many buses are needed?
a.
divide, then add
b.
subtract only
c.
multiply only
d.
divide only
Purpose: To assess knowledge of addition and subtraction of signed numbers in the bunny game.
Test
Knowledge
Description
Score(s)
Bunny
game
Conceptual
Given an arithmetic
problem, use virtual
joystick to simulate
movement of bunny
along number line.
Number correct,
misconception type
Arithmetic
solving
Strategic,
conceptual
Given an arithmetic
problem, type in
correct answer.
Number correct,
misconception type
From Moreno & Mayer (1999)
Bunny game
4 - -5 =___
-9 -8 -7 -6 -5 -4 -3 -2 -1 0
1 2 3 4 5 6 7 8 9
JUMP FORWARD
FACE LEFT
FACE RIGHT
JUMP BACK
Purpose: To assess knowledge of how plants grow in the design-a-plant game.
Test
Knowledge
Description
Score(s)
Design-aPlant game
conceptual
Select roots, stem, and leaves
for a plant to grow in a
specified environment.
Number correct
Given a plant with specified
roots, stem, and leaves,
describe the environment it is
best suited for.
Number correct
From Moreno et al. (2001)
Design-a-plant game
“Design a plant to live in an environment with low sunlight.”
[Select leaves, stem, and roots.]
“Describe the environment that would be best for a plant with
thick, small, think-skinned leaves; think, long, and dark bark stem;
and branching, shallow, and thin roots.”
Purpose: To assess knowledge of geology sonar detection in the profile game.
Test
Knowledge
Description
Score(s)
Profile game
strategic
Detect geological features
in a geology simulation.
Number correct,
solution time
From Mayer, Mautone, & Prothero (2002)
Section of earth
Draw a line here
Profile line
See its profile here
Purpose: Assess knowledge of how an electric motor works in the Dr. Phyz simulation.
Test
Knowledge
Description
Score(s)
Troubleshooting
problem
conceptual
List possible reasons
for motor to not
work when turned on.
Number of
correct answers
Redesign
problem
conceptual
Describe how to change
a motor for a certain
purpose.
Number of
correct answers
Principle
problem
conceptual
Explain why an event
occurs.
Number of
correct answers
Prediction
problem
conceptual.
Tell what would happen
for a specific set of actions.
Number of
correct answers
Action
problem
conceptual
Tell actions to take to make
the motor accomplish a
Number of
correct answers
specific goal.
From Mayer et al. (2003)
Troubleshooting problem
“Suppose you switch on an electric motor but nothing happens.
What could have gone wrong?
Redesign problem
“What could you do to increase the speed of the electric motor,
that is, to make the wire loop rotate more rapidly?
Principle problem
“Why does the wire loop move?
Prediction problem
“What happens if you connect the negative terminal to the red
wire and the positive terminal to the yellow wire?”
Action problem
“What can you do to reverse the movement of an electric motor,
that is, to make the wire loop rotate in the opposite direction?”
Purpose: To assess knowledge of an aircraft fuel system in the aircraft simulation.
Test
Knowledge
Description
Score(s)
Troubleshooting
problem
conceptual
List possible reasons for
fuel system to malfunction.
Number correct
Redesign
problem
conceptual
Describe how to change the
fuel system for a certain
purpose
Number correct
Prescription
problem
conceptual
Tell what would happen for
a specific set of actions
Number correct
From O’Neil et al. (2000)
Troubleshooting problem
“Suppose you try to transfer fuel from one tank to another but
nothing happens. What could have gone wrong?”
Redesign problem
“What could be done to make fuel transfer more reliable, that is, to
minimize the chances of something going wrong when fuel is
transferred from one tank to the next?”
Prediction problem
“What would happen if the ejector pump stopped working?”
Conclusion: Problem-solving performance in any domain depends
on the learner’s knowledge, which is measurable. Conceptual and
strategic knowledge can be assessed.
Assess conceptual knowledge by asking the learner to:
Specify what happens when a command is executed
Translate a word problem into an equation
Tell what numbers are needed to solve a word problem
Ask a learner to role play what happens for a numerical computation
Answering open-ended questions involving troubleshooting,
redesign, predicting, explaining, and prescribing
Assess strategic knowledge by asking the learner to:
Generate or follow directions
Describe the output of a procedure
Specify the operations needed to solve a word problem
Learn to play a new game
Make judgments