Storytelling Alice:
Presenting Programming as
a Means to the End of
Storytelling
Caitlin Kelleher
Carnegie Mellon University
Pittsburgh, PA
www.alice.org
1
Shrinking Pool of Computer
Scientists
2
There is a strong need for
computer scientists
• Three occupational groups are expected
to provide 75% of the new jobs between
2004-2014
– Health care practitioner and technical
occupations
– Education, training, and library occupations
– Computer and mathematical science
occupations
3
Projected Science and Engineering Job Openings
(new jobs and replacements)
2004-2014
Engineering
Information
Technology
Life Science
Physical
Science
Natural
Science
Managers
4
Other Countries Predict IT
Shortages Too
• Canada: the current 35,000 openings per
year will grow to 89,000 by 2010.
Currently, schools are producing < 7000
per year.
• UK: 50% drop in students since 2000 will
lead to hiring difficulties.
5
Computer Science supports
progress across many disciplines
6
Simply, we need more
computer scientists
…and more diverse computer
scientists
7
Broad reach creates a pragmatic
need for diversity
8
Lack of diversity can lead to
unintentional exclusion
9
Research goal:
Design programming
environments that motivate a
larger and more diverse group
of people to learn to program.
10
Related Work
11
Related Work
Simplify the language:
BASIC
12
Related Work
Simplify the language:
BASIC
Prevent syntax errors:
Cornell
Program Synthesizer
13
Related Work
Simplify the language:
BASIC
Prevent syntax errors:
Cornell
Program Synthesizer
Make programming
concrete:
Karel, Logo
14
Related Work
Simplify the language:
BASIC
Prevent syntax errors:
Cornell
Program Synthesizer
Make programming
concrete:
Karel, Logo
Construct programs with
graphical objects:
Lego Mindstorms
15
Related Work
Simplify the language:
BASIC
Prevent syntax errors:
Cornell
Program Synthesizer
Make programming
concrete:
Karel, Logo
Construct programs with
graphical objects:
Lego Mindstorms
Introduce new
programming paradigms:
Smalltalk, Pascal
16
No matter how easy something
is, people still need a reason to
want to do it.
17
Approach:
Develop programming
environments that present
programming as a means to an
end that the target audience is
motivated to pursue.
18
Solve the harder problem: make
programming motivating for an
under-represented group
• There are many important and underrepresented groups in Computer Science.
• Women are currently the largest.
19
Female CS pool is shrinking
quickly.
2004
1973
20
Critical Time: Middle School
• In 4th grade, the numbers of girls and boys
who like math and science are about
equal.
• By the end of 8th grade, twice as many
boys as girls are interested in math and
science.
21
Approach
Present programming as a
means to the end of storytelling.
22
Why Storytelling?
•
•
•
•
Given a little bit of time, most girls can
come up with a story they want to tell.
Stories are naturally sequential.
Stories provide girls an opportunity to
experiment with different roles, a
central activity during adolescence.
Non-programming friends can readily
understand and appreciate an
animated story.
23
Developing Storytelling Alice
24
Started with Alice 2
• Alice 2 is a programming environment targeted
at college-level intro CS students
– Separate learning of programming concepts and
syntax
– Enable novice users to create interactive 3D graphics
programs
– Formal results show that use of Alice increases
grades and retention of CS students in subsequent
Java-based CS courses.
• Alice 2 provides:
– drag and drop program construction that
prevents syntax errors
– 3D graphics and animation engine
25
Classic HCI Research Paradigm
Over 18 months, ~15 iterations
Formative Evaluation:
Observe middle school
girls (and some boys)
creating stories in Alice
Design and Implement:
Improve Storytelling Alice
26
Subjects: Formative Evaluation
• ~ 60 Campers at
STEM camps
• ~ 30 Local homeschooled kids
• ~ 120 Girl
Scouts from
within 1.5 hours
of Pittsburgh
27
Storytelling Alice Demo
28
Changes to Enable Storytelling
1. Add high-level animations and scene
support.
2. Create a story-based tutorial.
3. Provide a gallery of 3D characters that
inspire stories.
29
Changes to Enable Storytelling
1. Add high-level animations and scene
support.
2. Create a story-based tutorial.
3. Provide a gallery of 3D characters that
inspire stories.
30
Lesson Learned: two-person talk
aloud protocol is a poor match for a
creative process like storytelling
31
Storyboarding Process
1. Write the DVD box description
2. Flesh out each scene: setting, purpose,
and action
3. Create 6-9 frames/textual descriptions
for each scene.
32
Rhetorical Information Helps
• 3 sources for actions:
– Users’ descriptions of the action in a given
frame
– Differences between drawings of sequential
frames
– Users’ descriptions of the action in a scene
(occasionally)
33
People are more important than
objects
Kids very rarely
animate things.
In Storytelling Alice, humanoids and other nonbipedal characters are a different type and have
different methods than non-living “things”.
34
What people can do…
Generic Alice:
Storytelling Alice:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Move
Turn
Roll
Resize
Play Sound
Move to
Move toward, Move away from
Orient to
Turn to Face, point at
Set point of view to
Set Pose
Stand up
Move at speed, turn at speed, roll at
speed
Constrain to face, Constrain to point at
35
Say, think
Walk to, Walk offscreen
Walk
Sit On
Lie on
Kneel
Fall Down
Stand Up
Straighten
Look at
Look
Turn to face, Turn away from
Turn
Touch
Keep Touching
Most character and object
interactions can be created using
touch and keep touching
36
Character actions often motivate
use of programming constructs like
loops, methods and parameters
37
Changes to Enable Storytelling
1. Add high-level animations and scene
support.
2. Create a story-based tutorial.
3. Provide a gallery of 3D characters that
inspire stories.
38
Before:
After:
39
Click on the “trevor” tile to see what
methods trevor knows how to do.
40
41
Stencils enables users to complete tutorials
more quickly and with fewer errors.
Users of a Stencils tutorial:
• Learned as well as users of a paper
version of the same tutorial
• Made 270% fewer errors (p < .05)
• Completed tutorial 26% faster
(p = .057)
•
Made 88% fewer requests for
human assistance (p = .08)
42
Changes to Enable Storytelling
1. Add high-level animations and scene
support.
2. Create a story-based tutorial.
3. Provide a gallery of 3D characters that
inspire stories.
43
Provide custom animations that
require explanation in the story
Harold T. Wireton.crazy go nuts
Animations can be incredibly powerful in
helping kids to come up with a story idea.
44
Provide characters with clear roles
Hero or Villain
Authority Figure
45
Related Work
46
Programming Systems that enable
animation
•
•
•
•
•
ToonTalk (Kahn, 1996)
Squeak Etoys (Kay et al., 1997)
HANDS (Pane, 2002)
Scratch (Resnick et al., 2006)
Kids Programming Language (Morrison
Schwartz, Inc., 2006)
47
Present programming in a
motivating context
• Algoarena – programming sumo-wrestlers
to fight with each other
• Robocode – programming tanks to fight
with each other
• Rocky’s Boots – connecting logic gates to
solve puzzles
48
Virtual Family
49
Rapunsel
50
There are no formal results
showing that any of these
systems are (more) motivating.
51
Evaluating Storytelling Alice
52
Two Questions
• How much do users learn?
• How motivated are they?
53
Two Questions
• How much do users learn?
• How motivated are they?
54
Summative Evaluation
• 88 Girl Scouts
from within 1.5
hours of
Pittsburgh
• Compare
Storytelling Alice
with Generic
Alice
55
Evaluating Generic Alice vs.
Storytelling Alice
Experimental Group:
Control Group:
Tutorial
Build something to show
in Storytelling Alice
Tutorial
Build something to show
in Generic Alice
2 hrs, 15 min
Take programming quiz and attitude survey
Try Storytelling Alice
Try Generic Alice
Choose Generic or Storytelling Alice to take home
Show a world to everyone
56
30 min
Results
57
Similar Learning in Storytelling Alice and
Generic Alice
Mean Programming Quiz Scores
1
0.9
Mean Score
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Sequence
Do
Together
Simple
Loop
More
Complex
Loop
Generic Alice
58
Method
Call
Parameter
Storytelling Alice
World
Starts
Event
Since it is as educational,
which system do girls find
more motivating?
59
Types of Results
• What girls build
• What girls say
• What girls do
60
Three Activities in Alice
2: Editing Programs
1: Scene Layout
3: Running Programs
61
Users of Storytelling Alice spend 42%
more time programming
Average % Time Spent on Alice Activities
60.00%
p< 0.001
50.00%
p < 0.001
40.00%
Generic Alice
Storytelling Alice
30.00%
20.00%
10.00%
0.00%
Scene Layout
Editing Program
62
Running Program
Storytelling Alice motivates the reluctant
programmers
Scene Layout vs. Program Editing
100
% Time - Scene Layout
90
80
70
60
50
40
30
20
10
0
0
10
20
30
40
50
60
70
% Time - Editing Program
Generic Alice
Storytelling Alice
63
80
90
100
Generic Alice: Arbitrary motion
(28/45 = 62%)
A collection of characters that moves in seemingly arbitrary ways.
Sometimes whole characters move, sometimes individual body parts.
64
Generic Alice: Character Motion
(7/45 = 15%)
The characters put their arms at their sides and the girl on the
left waves hello.
65
Generic Alice: Story-like sequences
(7/45 = 15%)
A penguin moves to the lever, the lever turns and the
Christmas tree lights up.
66
Generic Alice: Dance routines
(3/45 = 7%)
The penguins turn, jump, and look in different directions in
sequence and in parallel.
67
Storytelling Alice: Relationship
Stories (22/43 = 51%)
A story about a boy named Dave who has been having relationships
with three different girls. They find out and kick his legs in retaliation.
The story ends with the statement “And that’s why you don’t cheat
on girls. It Makes Your Legs Fall Off!”
68
Storytelling Alice: Good vs. Evil
(9/43 = 21%)
A story in which the Big Bad Wolf
attempts to befriend the three
pigs in the hopes of eating them.
The pigs get scared and a ninja
appears to frighten the wolf away.
69
Storytelling Alice: other
(12/43 = 28%)
• 2 stories about finding lost dogs
• 2 stories depicting running or swimming
races
• 3 choreographed routines involving
cheerleaders or circus characters
70
Types of Results
• What girls build
• What girls say
• What girls do
71
Ease and Entertainment
• No significant differences between
Storytelling Alice and Generic Alice users:
– Alice’s ease of use
– Alice’s entertainment value
72
Users of Storytelling Alice have stronger
interest in future Alice use.
Mean Future Alice Use Scores
p < 0.05
3.5
3
Mean Score
2.5
2
Gener ic Alice
St or yt elling Alice
1.5
1
0.5
0
One time use
Use over the next month
Question
73
Interest in a future Alice
class
Users of Storytelling Alice were slightly
(but not significantly) more interested in
computer science
Mean Computer Science Interest Scores
1.5
Mean Score
1
0.5
Gener ic Alice
St or yt elling Alice
0
- 0.5
Create a w orld
Learn
you are proud
advanced
of
Alice features
Learn
Java/C++
-1
Question
74
Interest in HS Grow up to CS
CS class
Types of Results
• What girls build
• What girls say
• What girls do
75
Users of Storytelling and Generic Alice
prefer Storytelling Alice
Alice Version to Take Home
100.00%
p < 0.001
90.00%
80.00%
% of Users
70.00%
60.00%
50.00%
40.00%
30.00%
20.00%
10.00%
0.00%
Chose Storytelling Alice
Chose Generic Alice
Alice Version Selected
Generic Alice
76
Storytelling Alice
Users of Generic Alice are more likely to
show the world they build in 30 minutes.
Switching Versions
100%
p < 0.001
Percentage of Users
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Show non-main world
Alice Program
Generic Alice
77
Storytelling Alice
Users of Storytelling Alice are more likely to
sneak extra time to continue programming.
Sneaking Extra Time
p < 0.001
100%
Percentage of Users
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Show non-main world
Alice Program
Generic Alice
78
Storytelling Alice
Summary of Results
• Users of Storytelling Alice
– Spent 42% more time programming
– Were more than three times as likely to sneak
extra time to work on their programs (51% vs
16%)
– Express stronger interest in future Alice use
• Users of both Storytelling and Generic
Alice preferred Storytelling Alice
79
Future Work
80
Research goal:
Design programming
environments that motivate a
larger and more diverse group
of people to learn to program.
81
Other Demographics
• Evaluating and potentially adapting
Storytelling Alice for other groups
– Boys
– Minority students
82
The Real World
• Realistically, most schools will introduce
programming in a short unit (if at all)
• Organizations like the Girl/Boy Scouts or a
local YMCA will tend towards a small
number of sessions.
• Storytelling Alice needs to inspire and
enable kids to learn new material on
their own.
83
Challenge #1:
keep users coming back for more
84
Addictive Educational Software
• Choose and support intrinsically motivating activities.
• What extrinsic rewards can reinforce users’ interest in
educational content over long periods of time?
– Access to new content?
– Positive feedback from peers?
• How do we measure motivation over longer periods?
• Effective extrinsic reward schemes and
motivational measures can aid the
development of serious games.
85
Challenge #2:
motivate users to learn more
programming concepts
86
87
Motivating Use of More
Programming Concepts
• More expressive character animations
(especially facial and social), to motivate
methods and parameterized methods
• Controlling behavior of groups of movie extras to
introduce conditionals, iteration, and abstraction.
• Could contribute to systems that enable
non-technical users to quickly prototype
film pre-visualizations and video
games
88
Challenge #3:
enable users to teach themselves
89
Using Example Code
• Understanding code: what does this do?
• Re-using code: Copy and modify example
code for use in your own program
• Interaction techniques for understanding
and re-using example code may lead to
improved general-purpose
programming environments
90
Questions?
Caitlin Kelleher
caitlin@cs.cmu.edu
www.alice.org
91