Carolyn Penstein Rosé
Language Technologies Institute
and Human-Computer Interaction Institute
With funding from the National Science Foundation and the Office of Naval Research
For more information about my group:
http://www.cs.cmu.edu/~cprose
Acknowledgements
Iris
Howley
Hua
Ai
Dong
Nguyen
Elijah Mayfield
Rohit Kumar
 Nguyen, D., Mayfield, E., & Rosé, C. P. (2010).
An analysis of perspectives in interactive
settings, in Proceedings of the KDD Workshop
on Social Media Analytics.
 Kumar, R. & Rosé, C. P. (2010). Engaging
learning Hua
groups using Social Interaction
Iris
Strategies, In Proceedings of the North
Howley
American Ai
Chapter of the Association
for
Computational Linguistics.
 Howley, I., Mayfield, E. & Rosé, C. P. (to appear).
Linguistic Analysis Methods for Studying Small
Groups, in Cindy Hmelo-Silver, Angela
Dong
O’Donnell, Carol Chan,
& Clark Chin (Eds.)
International Handbook
of Collaborative
Nguyen
Learning, Taylor and Francis, Inc.
 Ai, H., Kumar, R., Nguyen, D., Nagasunder, A.,
Rosé, C. P. (2010). Exploring the Effectiveness
of Social Capabilities and Goal Alignment in
Computer Supported Collaborative Learning,
in Proceedings of Intelligent Tutoring Systems.
Rohit Kumar
Elijah Mayfield
Outline
 Motivation from Opinion Mining
 Theoretical framework from Rhetoric and Discourse
Analysis
 Study one: Political bias in a political discussion forum
 Study two: Goal orientation in chat based design
discussions
 Current Directions
Outline
 Motivation from Opinion Mining
 Theoretical framework from Rhetoric and Discourse
Analysis
 Study one: Political bias in a political discussion forum
 Study two: Goal orientation in chat based design
discussions
 Current Directions
Are product reviews
conversational?
KEY ASSUMPTION:
KEY ASSUMPTION:
language is
language
is a reflection
a reflection of the speaker’s
of perspective
the speaker’s
perspective
 Typical paradigm for sentiment analysis of product
reviews:
 Make a prediction based on text of single reviews taken out
of context
 Some evidence of group effects in product review blogs
based on numerical ratings (Wu et al., 2008)
Are product reviews
conversational?
KEY ASSUMPTION:
KEY ASSUMPTION:
language is
language
is a reflection
a reflection of the speaker’s
of perspective
the speaker’s
perspective
 Work towards weakening the assumption
 Sometimes use syntactic cues to reverse polarity on some terms
(Somasunderan & Wiebe, 2009; Wijaya & Bressan, 2008)
 Factoring out the effect of context rather than modeling it


Aggregation over all reviews posted by the same individual
Taking opinions of similar individuals into account (e.g.,
collaborative filtering)
 Falls short of modeling conversational aspects of
product reviews
Are product reviews
conversational?
 “After many MANY weeks of research, gathering
information from several sites, reviews etc I decided
that the Britax Boulevard was definitely the safest bet
available on the market. The things that sold me: All
the safety gadgets that other seats don't have like the
side impact wings, the HUGS system, the LATCH system
and 5 point harness and also the fact that it lasts up to
29Kg. “
Are product reviews
conversational?
 “I did most of my research on the net, picking my top
3 choices I went and had a look at them in the shops. I
looked at one the Graco Comfort Sport, the Britax
Boulevard and the Decathlon and Marathon seats. By
far it seems that Britax have the upper hand safely
wise on the market, many professional reviews and
crash tests agree on this so Britax was the clear choice
for us. “
Are product reviews
conversational?
 “I have the seat front facing in my Camry (2007) I
worried about the size of the chair from reading
other reviews but that is NO problem in my car, my
son has plenty of leg room and can see perfectly out
the window.”
Are product reviews
conversational?
http://www9.georgetown.edu/faculty/irvinem/theory/Bakhtin-MainTheory.html
Outline
 Motivation from Opinion Mining
 Theoretical framework from Rhetoric and
Discourse Analysis
 Study one: Political bias in a political discussion forum
 Study two: Goal orientation in chat based design
discussions
 Current Directions
Discourse and Identity
 Identity is reflected in the way we present ourselves
in conversational interactions
 Reflects who we are, how we think, and where we
belong
 Also reflects how we think of our audience
 Examples
 Regional dialect: shows my identification with where
I am from, but also shows I am comfortable letting you
identify me that way
 Jargon and technical terms: shows my identification
with a work community, but also shows I expect you
to be able to relate to that part of my life
 Level of formality: shows where we stand in relation
to one another
 Explicitness in reference: shows whether I am
treating you like an insider or an outsider
Discourse and Identity
 Discourse is text above the clause
level (Martin & Rose, 2007)
 A Discourse is an ongoing
conversation [type]
 Socialization is the process of joining a
Lakoff & Johnson, 1980
Discourse (Lave & Wenger, 1991; Sfard,
2010)
 We join Discourses that match our core
identity (de Fina, Schiffrin, & Bamberg,
2006)
 In moving from the periphery to the
core of a Discourse community, we
sound more and more like the
community (Arguello et al., 2006)
 A discourse is one instance of it
[token]
 All discourses contain echoes of
Lave & Wenger, 1991
previous discourses (Bakhtin, 1983)
Metaphors Structure our
Experience
 We describe arguments using
terms related to war
 Using a typical war ‘script’ to
structure a story about an
argument
 We orient towards arguments
as though they were wars
 Our conversational partner is
our opponent
 We may feel that we won or
lost
 We may feel wounded as a
result
Discourses, Frames, and
Metaphors
 Frame: A portion of a discourse belonging to
distinct Discourse
 Metaphor : One linguistic device that can be
used to define a set of discourse practices that
constitute a frame
 Topic models: a technical approach that makes
sense for identifying frames within a discourse
 A discourse could be drawn from a mixture of
Discourses
 Within the same conversation, we may wear a
variety of “hats”
 E.g., the same discourse with a co-worker may
contain exchanges pertaining to our relationship
as colleagues and others to our relationship as
friends
Model of Communication from
Rhetoric
Author
Implied
Author
Implied
Reader
Text
Effect
Reader
 Implied author: Communication
style is a projection of identity
 Impression management, not
necessarily the ground truth
 Implied reader: What we assume
about who is listening
 Real assumptions, possibly incorrect
 What we want recipients or
overhearers to think are our
assumptions
 Reader: may or may not understand
the text the way it was intended
Engagement: Social positioning in
conversational style
Author
Implied
Author
Implied
Reader
Text
Effect
Reader
 The message: Most contributions
express some content
 Implied author: How I phrase it says
something about my stance with
respect to that content
 Implied reader: Also says something
about what I assume is your stance
and my stance in relation to you
 Reader: The hearer may respond
either to the message or its
positioning
Engagement: Social positioning in
conversational style
Author
Implied
Author
Implied
Reader
Text
Effect
 Speaker 1: I want chocolate for dessert.
 Speaker 2: [you can’t have chocolate]
 Options with different implications about
author and reader


Reader




You can’t have chocolate.
You’re allergic to chocolate.
You’re allergic to chocolate, so eating it would
be a bad idea.
Your mom said you’re allergic to chocolate
Having chocolate might be a poor choice for
you.
Having chocolate might be a poor choice for
you for a great number of reasons.
Even Scientific Writing is Social and
Conversational
Author
Implied
Author
Implied
Reader
Text
Effect
Reader
 Implied author: Rhetorical style in
academic writing gives an impression
of who we are as researchers
 Implied reader: targeting writing to
community standards
 Abstracts and literature reviews
position us in research community
 Reader: research papers teach us
both about the content of our field
and its politics
Outline
 Motivation from Opinion Mining
 Theoretical framework from Rhetoric and Discourse
Analysis
 Study one: Political bias in a political discussion
forum
 Study two: Goal orientation in chat based design
discussions
 Current Directions
Bias Estimation
 Start with LDA model of politics dataset with 15 topics
 Then separate the texts into two collections, one left
affiliated, and one right affiliated
 We then have a Left model and a Right model
 We can then compute a rank for each word w in each topic t
in each model
 Intuition: a word is more distinguishing for a particular point
of view if it has a high probability within the associated model
and a low probability in the opposite model
 Bias(w,t) = log(rankright(w,t) + 1) – log(rankleft(w,t) + 1)
 The bias of a text is the average bias over the terms within
the text
 Left scores positive, right scores negative
Qualitative
Analysis
 Terror Language (Right): evokes emotional response to
thread of attack. Define target as evil and as a threat.
Provokes a defensive posture.
 Imperialist rhetoric (Right): racial prejudice, attitude of
superiority.
 Web of concern (Left): focus on opposition as
individuals with a culture and history, concern for
wellbeing of all people, focus on potential negative
effects of war
Quantitative Analysis
Right Bias
Left Bias
 Score of poster
 Score of quoted




message
Score of full post
Score of words
that appear in
both messages
Score of words
that appear only in
quoted message
Score of words
that appear only in
the post
Investigation of Quoting behavior
 Negative correlation between words only in quoted
message and words only in post (r=-0.1, p < 0.05)
 Positive correlation between score quoted words and
score of the whole post (r=0.18, p < 0.02)
 Score of words only in post are significantly more
reflective of the affiliation of the poster than that of the
author of the quoted message
 Similar result with score of words only in quote with
affiliation of author of quoted message
Investigation of Quoting behavior
Investigation of Quoting behavior
 Which words are quoted?
by pointing out the inflation of Saddam’s body count by neocons in an
effort to further vilify him and thus further justify our invasion we are
not DEFENDING saddam....just pointing out how neocons rarely let facts
get in the way of a good war.
So wait, how many do you think Saddam killed or oppressed? You’re
trying to make him look better than he actually was. You’re the one
inflating the casualties we’ve caused! Seriously, what estimates
(with a link) are there that we’ve killed over 100,000 civilians. Not
some crack pot geocities page either.
Thread level analysis
 Effect of initial post
 Correlation between score thread (without first post)
and first post = 0.210 (p<0.01)
 Effect of Prior posts
 Aggregate score of previous posts
 Difference in score of current post and average score of
user
 Small correlation (r=0.133, p < 0.01) indicating that
users talk more left than they usually do when previous
posts are left and similarly for right.
Overview of Findings
 Quotes from opposite point of view include the words
that are less strongly associated with the opposite
perspective
 Because of quotes, displayed bias shifts towards the
bias of the person to whom the message is directed
 Personal bias of the speaker is most strongly
represented by non-quoted portions of text
 The effect of a post extends past just the immediate
response
Outline
 Motivation from Opinion Mining
 Theoretical framework from Rhetoric and Discourse
Analysis
 Study one: Political bias in a political discussion forum
 Study two: Goal orientation in chat based design
discussions
 Current Directions
Collaborative Design Task
 Goal: Design a power plant
 Competing Student Goals:
 Power: Design a power plant that
achieves maximum power output
 Green: Design a power plant that has
the minimum impact on the
environment
 Competing goals encourages deeper
discussion
 Exploration of design space
 Explicit articulation of reasoning
> Experimental Design >Task
30
Classroom Study
 Chat room style
interaction
 ConcertChat
 106 students
 CMU undergrads in ME
 Classroom session
 During the semester
• Instruction Pretest 
Session  Posttest 
Questionnaire
31
> Experimental Design >Study Procedure
31
Experimental Design
• Social Behavior
– Frequent, Infrequent, None
• Goal Alignment
Frequent
Green
Infrequent
Green
None
Green
Frequent
Power
Infrequent
Power
None
Power
Frequent
Neutral
Infrequent
Neutral
None
Neutral
– Green, Power, Neutral
 Hypotheses
 Students will be more engaged with agents displaying
social behaviors
 Students will be sensitive to tutor goal orientation
 Interaction effect
> Experimental Design >Manipulations
32
An Example of Displaying Bias
Green Bias
 Green: What is bad about increasing heat input to the cycle is that
more waste heat is rejected to the environment.
 Neutral and Power: Increasing heat input to the cycle increases
waste heat rejected to the environment.
Power Bias:
 Power: What is good about increasing heat input to the cycle is
that more power output is produced.
 Neutral and Green: Increasing heat input to the cycle increases
power output produced.
> Experimental Design >Agent Design
33
Example of Social Behaviors
Adapted from Bales’ IPA (Bales, 195)
1. Showing Solidarity: Raises other's status, gives help, reward
Tutor: Let’s Introduce ourselves. My name is Avis.
Tutor: Be nice to your teammates!
2. Showing Tension Release: Jokes, laughs, shows satisfaction
Tutor: I’m happy to work with our team :-)
3. Agreeing: Shows passive acceptance, understands, concurs, complies
Tutor: m-hmm (showing attention)
34
> Experimental Design >Agent Design
34
Experimental Design
• Social Behavior
– Frequent, Infrequent, None
• Goal Alignment
Frequent
Green
Infrequent
Green
None
Green
Frequent
Power
Infrequent
Power
None
Power
Frequent
Neutral
Infrequent
Neutral
None
Neutral
– Green, Power, Neutral
 Hypotheses
 Students will be more engaged with agents displaying
social behaviors
 Students will be sensitive to tutor goal orientation
 Interaction effect
> Experimental Design >Manipulations
35
Measuring Student Bias
 Using a topic modeling tool – ccLDA [Paul and Girju, 2009]
Topic1
Corpus
Collection1 Topic 2 Collection2
Topic 3
36
> Experimental Design >Displaying Bias
36
Example of Extracted Topics
Background
Heat
quality
right
max
decrease
possible
goes
efficiency
need
gas
graph
say
natural
want
goal
fuel
Tmax
min
sounds
temp
going
friendly
turbine
kpa
mean
TOPIC 1
Green
11000
values
different
makes
larger
graphs
bit
green
large
kind
produce
hate
steam
team
step
solid
6574
split
bored
nat
geo
instead
happens
plant
love
Power
yah
blades
sir
dunno
kk
x85
rejected
guessing
starts
FINAL
life
helping
compromise
nd
depends
corresponding
teammate
stays
tmin
new
hard
sitting
afk
tmax500
bec
> Experimental Design >Displaying Bias
TOPIC 2
Background
power
decreases
nuclear
make
85
cycle
work
guess
high
pmin
want
pmax
wait
570
lower
green
40
tmax
Pmin
value
low
best
point
pick
environment
Green
low
500
12800
sort
1
tutors
effeciency
440
coool
ecofriendly
half
fun
105
Nuclear
sweeet
maximized
cooler
question
boy
6000
worked
creepy
Goes
16250
maxes
Power
generates
makes
085
different
7000
12000
qdot
becuase
decreasing
click
leads
liquid
gues
doubt
10790
meet
POWER
6574
DESIGN
transfer
hope
Qin
11000
discussion
km
37
Bias Measurement Metrics
 Max Topic-word bias: count the number of words in the list of
the N most strongly associated words, and take the maximum
across topics
 Average Topic-word bias: count the number of words in the list
of the N most strongly associated words, and take the average
across topics
 Weighted Topic-Word Average bias: Same but weight each word
by its association within the background model first
 All three measures highly correlated both for Green and for
Power perspectives
 Students in the Green condition got higher Green scores on
average than Power scores and vice versa in the Power condition
 Only statistically significant for the first two metrics
Measuring Influence
 Within pairs, the Green score of the Green student and the
Green score of the Power student were significantly
correlated
 Same story for Power scores
 Result consistent with analysis of Politics dataset
> Experimental Design >Displaying Bias
39
Operationalization of
Authoritativeness
 Negotiation coding
scheme (Martin & Rose,
2007, Chapter 7)
 Agreement on
K1/K2/Other
 .72 Kappa
 Coded all transcripts
from Infrequent Social
condition
 Authoritativeness score
= K1/[K1 + K2]
 Within pairs, one
Authoritative student
and one
NonAuthoritative
student
Balance Effect
• Alignment
• Align: Authoritative partner shares affiliation
with agent
• Neutral: Agent is neutral
• NoAlign: Non-Authoritative partner shares
affiliation with agent
• Affiliated agent has polarizing effect on
displayed bias
• Difference in bias scores was significantly higher
in conditions with affiliated agents
• Direction of polarization depends on alignment
• Balance Effect
• Authoritative student shows less of his own
bias when he’s in the minority
• NonAuthoritative student is less nonauthoritative when he’s in the minority
Overview of Findings
 Topic models can display differences in goal
orientation in chat data
 Confirmation of influence of partner speech on
displayed bias
 Complex relationship between personal orientation,
authoritativeness, and ingroup/outgroup effects
Outline
 Motivation from Opinion Mining
 Theoretical framework from Rhetoric and Discourse
Analysis
 Study one: Political bias in a political discussion forum
 Study two: Goal orientation in chat based design
discussions
 Current Directions
Current Directions
 Full circle for opinion mining
 Continuing to operationalize multiple dimensions of
relational codes
 Howley, I., Mayfield, E., & Rosé (to appear). Linguistic Analysis
Methods for Studying Small Groups, in Hmelo-Silver, O’Donnel,
& Chan (Eds.) International Handbook of Collaborative
Learning, Taylor & Francis, Inc.
 Collaboration with Bob Kraut: investigating how exchange
of social support is reflected in relational codes
 Collaboration with Bhiksha Raj: investigating evidence of
relational codes like Negotiation in speech
 CHI submission in progress: Analysis of effects of bullying
behavior on distribution of relational codes and learning
 New grant!: studying the emergence of leadership in ad-hoc
teams
Computational Models of
Discourse Analysis
 Focus on literature from the field of Discourse Analysis
 Investigating issues such as Conversational Structure,
Attitude, Perspective, Persuasion and Positioning
 Critical reflection on the state-of-the art in language
technologies
 Hands-on programming assignments, fun contests
Carolyn Penstein Rosé
http://www.cs.cmu.edu/~cprose
cprose@cs.cmu.edu
Gates-Hillman Center 5415