Motivation
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Agent Supports Discussions
Past
discussions
accumulative
data
Should performance enhancing
Update be allowed?
drugs
Current
discussions
Capital punishment?
Agent
Offer arguments
Trial by jury?
=Obtains information
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Argumentation Theory?
 Extensions?
 People do not reason logically.
 People differ in their
 Validity values?
 Justification value?
Dung
Wyner
Reed
argumentation choices.
 There is temporal nature of
argumentation.
Cayrol
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Parsons
Giacomin
Argumentation Theory?
Data Collection of 6 fictional Cases
 64 participants from Amazon Turk;
 age average: 38.5
 21 females; 17 males
 3 with Phd
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Argumentation Theory?
SUV
Safe
Too
expensive
8%
High
taxes
33%
-0.23
35%
Taking
Taking out
out
-0.33
aa loan
loan
High
interest
24%
Arvapally et al, 2012
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Argumentation Theory?
Transcription of Real Discussions
 Penn TreeBank Project (1995) conversation database:
 CAPITAL PUNISHMENT (33)
 TRIAL BY JURY (31)
Percentage of arguments from extensions
less than 30%
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Formal Model
 Argument (a)
 Argumentation Framework (AF)
 Short text
 A (Arguments)
 Attributes (m)
 R (Attack relation)
 S (Support relation)
 Deliberation (D)
 Takes place under some AF
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Argument’s Attributes
 Arguments Attributes:
 Justification.
 Psychological attributes.
 Relevance.
SUV
Safe
Too expensive
High taxes
Taking
out a loan
High
interest
Ariel Rosenfeld and Sarit Kraus, 25 November 2014
9
Prediction of Argument Choice
 Prediction features:
 Deliberation-context features.


Last given arguments (2 by each party).
Who said the last word.
 Deliberant features.


Average values for selected arguments.
Proneness.
Ariel Rosenfeld and Sarit Kraus, 25 November 2014
10
Prediction of Argument Choice
 Features:
 Justification
 Psychological features
 Deliberation-based features (the current state of the
deliberation)
SUV
 Relevance
Too
expensive
Safe
High
taxes
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Taking out
a loan
High
interest
One Time Argumentation Prediction
Experiment of 6 fictional scenarios
 The subject was asked to choose between 4 options.
 64 participants from Amazon Turk
SUV
Too
expensive
Safe
High
taxes
Taking out
a loan
High
interest
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Predicting Arguments
Given 5 choices of a subject, calculate the average of each feature, and
predict the 6th one.
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Culture based?
 78 Computer Science students.


CS-77% > AT 72%
exactly the same features as AT
 Can learn from one and predict to the other.


CS -> AT : 69% accuracy
AT -> CS: 76% accuracy
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Transcription of Real Discussions
 Capital punishment:
 33 examples
 AF of 33 nodes
 Trial by jury
 31 examples
 AF of 26 nodes
 All arguments appeared in 4 examples
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Capital Punishment
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Recommendation Policies
 How should the suggestions be presented?
 Should we suggest the predicted arguments?
 How to maintain a good hit-rate while offering novel
arguments?
 What will be beneficial for each user?
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Data collection
 Collected 72 chats (144 participants) on “influenza
vaccinations” using a structured chat.
 Half of the participants declared their opinion (before
the chat) and were coupled with the opposing view.
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Influenza Vaccinations
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Agents
 PRD: Prediction [17 chats]
 REL: Relevant [17 chats]
 WRL: Weakly related [17 chats]
 PRH: Prediction (2) + Relevant (1) [17 chats]
 TRY: Theory [17 chats]
 RND: Random [17 chats]
 204 participants overall.
Ariel Rosenfeld and Sarit Kraus, 25 November 2014
20
Normalized Acceptance Rate
 Prediction + Relevant selections distribution:
 65% prediction, 35% relevant.
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
User Satisfaction
 Prediction and Relevant agents outperform Weakly
related, Random and Theory agents.
 Their combination seems to be even better.
(significant using Fisher’s exact test)
Agent
Neutral
Positive
V. Positive
Prediction
5
10
2
Relevant
5
12
0
Weakly related
16
1
0
Prediction + Relevant
0
12
5
Theory
13
3
1
Random
14
3
0
Ariel Rosenfeld and Sarit Kraus, 25 November 2014
22
Conclusions (Prediction)
 Based on our experiments with over 400 human
subjects we conclude;
1.
2.
3.
Incorporating ML in argumentation is needed for
investigating argumentation in the actual world.
Combining the Relevance notion in abstract
argumentation theory should provide it additional
predicative strength.
Other aspects of argumentation besides
acceptance/validity should be explored.
Ariel Rosenfeld and Sarit Kraus, 25 November 2014
23
Conclusions (Policies)
 Even though the prediction model yields limited accuracy
using it provides solid policies.
 Combining prediction and relevance-based heuristic is
beneficiary for most users, this combination covers a larger
variety of users and provided high scores.
 Taking the human limitations and biases into
consideration is a good practice for application
development.
Ariel Rosenfeld and Sarit Kraus, 25 November 2014
24
Current Work –
Personalized Agent
 Repeated = same participant, different topics.
 What if we let the agent “learn” from previous chats?
 Could we do better?
 User-modeling for over-time multiple uses of the
system.
 Can we learn from past chats to future ones?
 Could we explore different policies over time?
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015
Thanks!
 Ariel Rosenfeld:
arielros1@gmail.com
 Sarit Kraus:
sarit@cs.biu.ac.il
Ariel Rosenfeld and Sarit Kraus ,AAAI-15 @ Austin, TX USA. January 2015