Carolyn Penstein Rosé
Carnegie Mellon University
Language Technologies Institute and
Human-Computer Interaction Institute
School of Computer Science
Social and
Communicative
Factors in
Learning
and
LightSIDE
Bringing two research
communities together….
Computer Supported
Collaborative Learning
Classroom Discourse
and
Teacher Professional Development
• Joint work with Lauren Resnick at the Learning Research and
Development Center
• Partnership with Pittsburgh Public Schools 9th grade biology
• District wide teacher professional development about classroom
discussion facilitation
• Student preparation for discussions using computer supported
collaborative learning
• Exciting finding: Preparing students with CSCL activities prior to
whole group classroom discussion has a facilitating effect on the
teacher (over 1 standard deviation effect!)
2
Automated Assessment
http://stateimpact.npr.org/ohio/2012/06/08/the-pros-and-cons-of-using-computers-to-teach-students-how-to-write/
• LightSIDE text mining tool
bench
– Http://www.cs.cmu.edu/~emayfiel/side.html
• Assessment of essay answers
• Assessment of student behavior
– Estimates of self-efficacy (Howley
et al., 2011)
– Estimates of effective discussion
behavior (Joshi & Rosé, 2007;
Rosé et al., 2008; Mu et al., 2012)
– Measures of teacher facilitation
behavior
3
Accountable Talk
(O’Connor, Michaels, & Resnick)
Eddie: Well, i don't think it matters what order the numbers are in. You
still get the same answer. But three times four and four times three
seem like they could be talking about different things.
Teacher: Rebecca, do you agree or disagree with what Eddie is saying?
Rebecca: Well, I agree that it doesn't matter which number is first,
because they both give you twelve. But I don't get what Eddie means
about them saying different things.
Teacher: Eddie, would you explain what you mean?
Eddie: Well, I just think that like three times four can mean three groups
of four things, like three bags of four apples. And four times three
means four bags of three apples, and those don't seem like the same
thing.
Tiffany: But you still have the same number of apples, so they are the
same!
Teacher: OK, so we have two different ideas here to talk about. Eddie
says the order does matter, because the two orders can be used to
describe different situations. So Tiffany, are you saying that three
times four and four times three can't be used to describe two different
situations?
VMT-Basilica
** Students learn up to 1.25 standard deviations more when interactive
support is provided in the environment.
Core Research Focus
• Identify conversational
interactions that are
valuable for learning
• Automatic conversation
analysis
– Facilitates learning research
– Automates assessment of
group processes
– Enables context sensitive
triggering of support
• Interactive support
technologies
6
Operationalization of
Productive Talk for Learning
• Drawn from multiple frameworks
valuing sociocognitive engagement
– Students explicitly display
their reasoning
– Students orient their
contributions towards
previous contributions
– Students increase in their ownership of
their ideas
• Made precise through constructs from linguistics
– Integrative book chapters (Sionti, Ai, Rosé, & Resnick, 2011; Howley,
Mayfield, & Rosé, 2012)
7
Learning Results
• Students in accountable talk classrooms (where
transactivity is elicited) learn more than students
in a traditional classroom (O’Connor, Michaels, &
Resnick, in preparation)
• Transactivity correlates with learning (Joshi & Rosé,
2007; Kumar et al., 2007)
– Consistent with results in connection with elaborated
explanations (e.g., Webb, Nemer, Zuniga, 2002)
• Collaboration support that increases transactivity
increases learning (Wang et al., 2007)
8
Historical Perspective…
• Socratic tutoring: directed lines of reasoning
– Evidence that socratic tutoring is more beneficial than
didactic tutoring (Rosé et al., 2001a)
• Socratic style implemented in Knowledge
Construction Dialogues (KCDs)
–
–
–
–
General attempt to model effective human tutoring
Hierarchical structure: adaptive to student needs
Used to elicit reflection
First used to support individual learning in Physics
• Atlas-Andes (Rosé et al., 2001)
• WHY-Atlas (Rosé et al., 2003; Rosé & VanLehn, 2005)
Empirical Foundation for
CycleTalk
• Human tutoring not always better than noninteractive support (VanLehn et al., 2007)
– Focus shift to capturing what it is about interaction that
is effective for instruction
• Human tutors guide students towards
opportunities for reflection (Rosé & Torrey, 2004)
• Human tutor support by effective tutors is
significantly better than hint based support (Rosé
et al., 2005)
Towards enriching the interaction…
• Student interaction with dialogue agents lacks
elaboration (Rosé et al., 2003)
• Students expect to behave differently with agents
(Rosé & Torrey, 2004)
• Students can benefit from working with another
student, even in the absence of computer
scaffolding (Gweon et al., 2006; Kumar et al., 2007)
11
Monitoring Collaboration with
Machine Learning Technology
Download tools at:
http://www.cs.cmu.edu/~emayfiel/side.html
Behavior
Labeled Texts
Labeled Texts
LightSIDE
Unlabeled Texts
<Triggered
A Model that can Label More Texts
Time
Intervention>
Using LightSIDE
Keep this picture in mind…
• Machine learning isn’t magic
• But it can be useful for
identifying meaningful patterns
in your data when used
properly
• Proper use requires insight into
your data
?
Error Analysis
VMT-Basilica
** Students learn up to 1.25 standard deviations more when interactive
support is provided in the environment.
Computer Supported Learning in
Large Classroom Studies
• University Level
– Freshman Engineering
– Calculus
– Sophomore Thermodynamics
• High School
– 9th Grade Biology
– Earth Sciences
• Middle School
– Engineering Outreach
– 6th grade math
Second-Year Thermodynamics
Project
• Sophomore Thermodynamics Course
– Rankine Cycle Design Competition
• Part of a Thermodynamic Cycles Lab (1 session)
• Students Work in Teams of 2 with Their Partner in Another Part of the Room
• Instructed on Effects of Changing System Variables (Temperature,
Pressure) on System Output (Power, Waste Heat)
• Goal: Increase Efficiency within Practical Constraints
• Software Package is CyclePad (Developed at Northwestern University
(Forbes et. al. 1999) to Allow Students to Construct and Analyze a Variety of
Thermodynamic Cycles)
Thermo Collaborative Task
• Goal: Design a power plant based
on the Rankine Cycle paradigm
– Each pair turns in exactly one design
• Competing Student Goals:
– Power: Design a power plant that
achieves maximum power output
• Motivated by economic concerns
– Green: Design a power plant that has
the minimum impact on the environment
• Motivated by environmental concerns
Increasing Conceptual Depth
• Student: so i think the best thing to go with first is to
increase the pressure entering the connector
• Tutor: You might want to review what you saw earlier in
the sensitivity analysis investigating the effect of
changing P-max on steam quality and efficiency, from
the top half of page 13 in your book.
• Tutor: type: HELP WITH CHANGEPMAX if you want to
discuss it with me.
• Student: help with changepmax
• Tutor: Remember the sensitivity analysis you have
completed where you manipulated P-Max and observed
it's effect on the efficiency of the cycle. You can refer
back to Figure 11 on page 13 of your booklet, which …
Example Interaction
In vivo studies
Pretest
Small Group
Activity
(Experimental
Manipulation)
Post-Activity
test
Whole class
Discussion
Post-Discussion
test
 3 in vivo studies in 9th grade biology
 Other similar studies in math, freshman engineering, thermodynamics, and chemistry
 Online small group activities, support from
Conversational Computer Agents
Example Intervention: Revoicing Agent
Auto Predicted AT
Teacher: Accountable Talk
Comparing Years 1 and 2
R = .45
R = .36
Average Student
Words per Turn
Student talk
Comparing Years 1 and 2
R = .59
R = .18
Current Directions
• Continuing to investigate social considerations
for integrating dialogue agents with groups
– Investigating how motivation orientation interacts with
treatment
• Working with groups larger than pairs (Gweon et
al., in press)
– Monitoring collaboration quality from speech
– Challenges of multi-party conversation analysis
• Multiple interwoven threads (Rosé et al., 1995; Wang et al.,
2008a,b)
27
Thank
You
!!!
Funding: The Office of Naval Research and the National Science Foundation
28