Proceedings of the 7th Annual ISC Graduate Research Symposium
ISC-GRS 2013
April 24, 2013, Rolla, Missouri
Ravi Arvapally
Department of Computer Science
Missouri University of Science and Technology, Rolla, MO 65409
IDENTIFYING OUTLIER OPINIONS IN AN ONLINE INTELLIGENT ARGUMENTATION
SYSTEM FOR COLLABORATIVE DECISION SUPPORT
ABSTRACT
The intelligent argumentation system allows stakeholders to
post their decision issues, alternatives and exchange their
arguments over the alternatives posted in an argumentation tree.
In an argumentation process stakeholders have their own
opinions and very often these opinions might be contrasting
and conflicting with others in the decision making group.
Some stakeholders have opinions which might be outlier with
respect to the mean opinion of the group. In this article, we
present a unique approach to identify stakeholders with outlier
opinions in the argumentation process. Our approach identifies
outlier opinions both based on a stakeholders’ opinion as well
as the collective opinion on a stakeholder’s opinion. The
decision maker and other participating stakeholders have the
opportunity to explore the outlier opinions existing in their
group both from an individual’s perspective as well as from the
group’s perspective. In a large argumentation tree, it is often
difficult to identify stakeholders with outlier opinions manually.
We also presented an experiment to evaluate our proposed
method.
1. INTRODUCTION
In a collaborative dialog process, stakeholders exchange their
views and opinions with reasoning. In the dialog process as
stakeholders exchange arguments, some change their opinion,
some strengthen, and some weaken their opinions [18]. It is
important to consider the opinion of each stakeholder in the
argumentation process for collaborative decision support.
In a collaborative decision making group, argumentation is a
crucial step and each stakeholder has unique preferences, and
opinions in the argumentation process. In a decision making
group, some stakeholders approach the problem very uniquely
and their opinions are farther away from any another
stakeholder in the opinion dimensionality. We refer those
opinions as the outlier opinions, since they are very different
from the individual opinions of the group. In face-to-face
discussions, one might be able to identify his peers with outlier
opinions.
By identifying the outlier opinions, the decision maker can
closely investigate the arguments posted by that stakeholder,
and consult that stakeholder to get in to more discussions. This
can refine the opinions of the group and help them develop
consensus within their group. It is the responsibility of the
decision maker and the decision making group to understand
the underlying semantics of the outlier opinion. This problem
is new to the domain of the argumentation systems. Few
researchers have worked on identifying extreme opinions or
ideas on Web-blogs and other social media such as YouTube
[3].
Earlier based on the similarity measurement, a framework was
developed to identify the polarization groups in argumentation
system [2]. But the framework in this section considers the
dissimilarity of each stakeholder’s opinion with the mean
opinion of the group and generates the ranked list of
stakeholders based on their dissimilarity values.
Our framework carries out two different methods in identifying
outlier opinions. First, the outlier opinions are identified based
on the aggregate opinion of a stakeholder which is computed
from his arguments. In the second method, the collective
determination score received by each stakeholder is computed
and the outlier detection algorithm is used. The results
produced by these two methods are later analyzed and
compared. We used a simple distance based outlier detection
algorithm to identify the outliers and inliers opinions in the
intelligent argumentation system. In the argumentation process
not only those viewpoints of a majority of participants need to
be analyzed but also those unique viewpoints from minority of
participants. Outlier opinions can accelerate new discussions
and possibly refine opinions of participating stakeholders. This
information can help the decision maker in taking more
appropriate actions during the decision making process. The
next section presents an update on the related literature work,
followed by a brief background on the intelligent
argumentation system. The opinion dissimilarity framework for
identifying outlier opinions is presented which is followed by
the experiment and conclusion of the article.
2. LITERATURE REVIEW
The following sub-section presents a brief introduction
about computer supported argumentation systems which is then
followed by a brief literature on outlier detection algorithms.
2.1. Computer Supported Argumentation Systems
A computer supported argumentation system allows
stakeholders to post their alternatives over issues and exchange
of arguments over the alternatives. An argumentation system is
1
developed based on a model. Some systems are developed
based on formal models, and some on informal models. Our
intelligent argumentation system is based on an informal
model.
Araucaria [1], Belvedere [20], ConvinceMe [5], and
Debatebase [6] are some of the argumentation systems that are
developed to support argumentation among students in
education. While some systems such as Carneades [30] are
developed considering the application in law and justice.
Several of these systems support basic support for constructing
argument trees or well known as argument diagramming. The
tradeoff between these systems is primarily based on two
different criteria, ability to express well and usability of the
system. Systems based on the formal argumentation models
provide great ability to express. This is a great benefit to the
participating stakeholders. At the same time these systems are
less usable in reality. On the other hand, argumentation
systems based on the informal models provides great usability
but less expressing ability. CoPe_it! [10, 11] is one of the good
web-based argumentation systems developed in recent times.
This system provides different projections. The basic projection
allows a group to discuss and construct trees. As one goes from
one projection to the other, the formality of the underlying
models gets stronger. This special feature allows for great
expressiveness and more usable at the expense of switching
projections.
Our major interest lies in the argumentation systems
developed based on informal models and specially built for
collaborative decision support. One could develop algorithms
for processing arguments in informal model based
argumentation systems and provide decision support. The
Synergy system [28], Deliberatorium [12, 13], HERMES [9],
CoPe_it! [10, 11] and the intelligent argumentation system [16,
17] are more easy to use in reality. The Synergy system is
relatively a newer one. The MIT Deliberatorium allows a
group to exchange arguments and construct argument maps.
The researchers are currently working on implementing several
deliberation metrics [14] that encourages a group to carry out
more discussions and it provides sort of social dynamics
information that exists in a group to the group stakeholders.
Several important social dimension aspects exist in face-toface meetings and discussions. In group discussions and
debates, with analytical ability one can identify people with
outlier opinions. This identification helps others in the group to
engage in more discussions to overcome the conflicts in
opinions. Sillince and Saeedi [23, 24] discovered that several
important aspects of face-to-face meetings and discussions such
as social, emotional and symbolic requirements are missing in
the current group decision support systems and computer
supported argumentation systems. Our effort is to incorporate
this special feature into the intelligent argumentation system.
We are not aware of any computer supported argumentation
system that identifies stakeholders with outlier opinions in
group discussions.
2.2. Outlier Detection Algorithms
According to Hawkins “Outliers are observations which
deviate significantly from other observations as to arouse
suspicion that these are generated by a different mechanism
[8]”. Outlier detection techniques are very crucial in the data
mining applications. The outlier detection algorithms have
been well applied and used in the complex network systems.
Gogoi et al., in their article classified the outlier techniques
broadly in to distance-based, density-based, and machine
learning based techniques [7]. In this article, we intend to use
the outlier detection techniques in the argumentation network.
There are several supervised outlier detection techniques and
unsupervised outlier detection techniques.
For using supervised outlier detection techniques, one
should have training data, in our case it would be more
appropriate to use unsupervised outlier detection techniques.
There are several other distance based statistical approaches to
identify outlier detection techniques and there are also other
soft computing outlier detection techniques using fuzzy
systems. Several outlier detection algorithms such as K-nearest
neighbor’s method [21], local-distance based methods [4, 25],
angle based outlier detection [15] and density based outlier
detection algorithms [19] are used in several applications so far
but not in argumentation systems.
3. INTELLIGENT ARGUMENTATION SYSTEM
This section presents the elements of an argumentation tree and
the argumentation reduction fuzzy inference system. The
intelligent argumentation system supports several elements in
its argumentation tree. A stakeholder can post decision making
issue under a project, while a project node can have several
child issue nodes. On issue node, one can attach several
alternatives pertaining to the decision making issue. Figure 1
presents the position dialog graph that represents the position
sub-tree in an argumentation tree. The position in Figure 1 is an
alternative solution pertaining to the decision making issue.
Arguments can be either directly or indirectly associated with
the position. A stakeholder can post arguments supporting or
attacking a position or an argument in an argumentation tree.
When a stakeholder posts an argument, he is responsible to
post the degree of strength of his argument along with the
argument text. This degree represents the strength and
association with its parent argument. The strength of an
argument ranges from -1 to +1. If a stakeholder’s argument is
attacking its parent node, then the stakeholder can select a
strength value from -0.1 to -1.0. If a stakeholder’s argument is
supporting its parent node, then the stakeholder can select a
strength value from +0.1 to +1.0. If an argument is neutral or
indecisive, then a stakeholder can post zero strength. A
stakeholder can support or attack other arguments or his own
arguments. An individual can post evidences supporting his/her
arguments. Figure 2 presents a snapshot of the intelligent
argumentation system.
The argumentation reduction fuzzy inference system
performs inference on the argumentation tree to compute the
favorability (support) of an alternative in the decision making
2
group. The inference system performs inference and ensures
that at the end all the arguments in a tree are directly associated
to its’ respective alternatives. This inference process is not
visible to stakeholders.
alternative. This inference system will be used further in the
framework section to compute a stakeholder’s opinion. For
more information about the system please refer to some papers
on intelligent argumentation system [16, 17].
4. FRAMEWORK
Figure 3 presents the proposed framework for identifying and
assessing outlier opinions in an argumentation process from
both the individual and the collective viewpoint.
An
argumentation tree is provided as input for processing and
analyzing arguments. The framework will generate the ranked
lists of outlier opinions. The first sub-section presents how the
outlier opinions are identified from the individual viewpoint
(method 1) and the second sub-section presents from the
collective viewpoint (method 2).
Figure 1. Position dialog graph
Figure 2. A snapshot of the intelligent argumentation
system
The inference process is performed using the strengths of
the arguments and the inference rules. When an argument has
to undergo inference process, the strength of the argument and
the strength of it’s parent argument are provided as inputs to the
argumentation reduction fuzzy inference system. The inference
process is conducted based on the general fuzzy heuristic rules.
The following examples are some of the general fuzzy heuristic
rules.
• If argument B supports argument A and argument A
supports position P, then argument B supports position P.
• If argument B attacks argument A and argument A
supports position P, then argument B attacks position P.
Strong Attack (SA), Medium Attack (MA), Indecisive (I),
Medium Support (MS), and Strong Support (SS) are the five
fuzzy linguistic labels. These labels are based on the fuzzy
membership functions and are not visible to the participants
during the inference process. Based on the combination of these
five variables, twenty fuzzy inference rules are developed.
After the inference process, the aggregation on strengths of the
arguments will be performed to compute the favorability of an
4.1. Method 1 – Individual Opinions
Step 1 –In the first step of this framework, the system carries
out the argumentation reduction inference process using the
fuzzy argumentation reduction inference system on the
argumentation tree. This process is carried out to compute a
stakeholder’s aggregate opinions. In the argumentation tree the
arguments are either directly associated or indirectly associated
with their respective positions. After the argumentation
reduction inference process, all arguments are directly
associated to their respective positions [16, 17].
In Figure 4, stakeholder S2 has contributed three arguments
under position 1. While one argument is directly associated
with position 1, and the other two are associated with the
arguments posted by stakeholder S1. The association between
(Arg1, position 1) and (Arg4, Arg1) are considered for using
the appropriate fuzzy inference rules. Based on the suitable
fuzzy rule, the Arg4 is reduced level by level such that it is
directly associated to position 1. The same procedure was
conducted for Arg6. The system ensures that all arguments
posted by a stakeholder are directly associated to an argument.
The argument based fuzzy inference system then reassesses the
strengths of the arguments based on the inference rules. The
new strength that an argument is assigned is relative to the
solution alternative. See Figure 5 for the argumentation tree
after the fuzzy inference process.
The favorability of
stakeholder S2 for position 1 is the aggregate of the argument’s
strength: Arg2, Arg4, and Arg6 (see Figure 5).
Step 2 – In the second step, the strengths of the arguments
posted by a stakeholder are aggregated to compute the overall
favorability of a stakeholder for that alternative. There by the
system computes the stakeholder’s favorability for each
position to compute a stakeholder’s favorability for that
position. The favorability of a stakeholder for all the
alternatives is represented as a vector, called as opinion vectors.
Each element in the vector presents the favorability of
stakeholder for a position.
Step 3 – The opinion vectors can be represented in the opinion
dimensionality. The opinion vectors are normalized using the
3
min-max normalization technique (Eq. 1). Min A and max A
represent the minimum and the maximum values in the data set
respectively. While we assign new_max A to +1 and new_min
A to -1, since we want the new data to be normalized within the
range of -1 and +1.
v' 
v  min A
(new _ max A  new _ min A)  new _ min A
max A  min A
(1)
4.2. Method 2 – Collective Opinions
Step 1 – On each argument in the argumentation tree, the
argumentation reduction inference process is carried out using
the fuzzy argumentation reduction inference system. Each
argument’s collective determination value will be derived, see
Figure 6. In method 1, the argumentation reduction process was
carried out on an alternative of a tree. But here, the
argumentation reduction process is carried out on each and
every argument to compute the favorability of each argument
from other arguments in the tree. In Figure 6, A1, A2, A3, A4,
A5, and A6 represents arguments.
Figure 5. Argumentation tree after argument inference
Algorithm 1 - Distance based dissimilarity
algorithm
Figure 3. Framework for identifying outlier opinions in the
argumentation process
Input: Opinion vectors
Output: Ranked list of opinions based on the farthest from
the mean opinion of the group.
Step1 – Compute the mean vector (X) of the input
opinion vectors (Y).
Step2 – Compute the Euclidean distance (Eq2.) from the
mean opinion vector and all input opinion vectors.
Step3 – Sort and generate the ranked list based on the
distance between opinion vector and mean vector.
𝐷 𝑋, 𝑌
= 𝑋1 − 𝑌1 2 + 𝑋2 − 𝑌2 2 + 𝑋3 − 𝑌3 2
2
Figure 4. Sample argumentation tree before argument
inference
Step 4 – The distance based outlier detection algorithm
(Algorithm 1) will be applied on the vectors to generate a
ranked list based on the opinion vector’s distance with the
mean opinion of the group. We used a simple distance based
outlier detection algorithm on our data (Algorithm 1).
Step 5 – Analyze results – The ranked list generated in step 4 is
used for analyzing the results. The top stakeholder (opinion) in
the list presents the opinion which is farthest from the mean
opinion of the group. The last element in the generated list
presents the opinion which is closest to the mean opinion of the
group.
Step 2 – In this step, the collective determination values of all
arguments of each stakeholder under a position are aggregated.
The aggregate collective determination of a stakeholder on all
positions is derived and represented as a vector. This process
will be carried out for all the arguments and stakeholders in the
argumentation process. Each element in a vector represents the
aggregate collective determination that a stakeholder’s
arguments have received under a position.
Step 3 – The vectors are then normalized using min-max
normalization technique (Eq. 1) to attain consistency in the
data. This determines the total collective determination
received by a stakeholder under a position. This process is
4
carried out for all stakeholders across all positions posted under
an issue in the argumentation tree.
Step 4 – The distance based outlier detection algorithm
(Algorithm 1) that is presented in algorithm 1 is also used here
to generate a ranked list of inliers and outlier opinions.
inliers opinions using the results in Table 1. The information
from Table 1 can be used to see the ranking and the overlap of
rankings in the outlier opinion ranks.
Stakeholder S12’s opinion is ranked as outlier number seven
from both the individual method as well as the collective
determination method. There are multiple cases one can
analyze, for example, ‘From an individual’s perspective, his
opinion is outlier, but from the collective perspective, his
opinion is not an outlier’.
Table 1. Ranked list of outliers based on both individual and
collective methods
Ranked list of outliers Ranked list of outliers
based on the stakeholders’ based on the collective
individual opinions.
determination values.
S18, S19, S21, S23, S9, S17,
S12, S20, S16, S2, S8, S5, S4,
S14, S15, S1, S10, S11, S22,
S3, S24, S6, S13, S7
Figure 6. Computing collective determination of arguments
Step 5 – The first opinion in the ranked list presents the opinion
(outlier) which is farthest from the mean opinion of the group.
The last element in the generated list presents the opinion
which is closest to the mean opinion of the group. The list
generated here is from the group’s perspective.
5. EMPIRICAL EVALUATION
In this section we present the empirical evaluation of the
proposed method. The data in our experiments is from the
experiments that were conducted in spring 2010 [22]. The
decision issue was about the selection of software metrics
program for a large scale organization for the given project
described in the case study. Comprehensive metrics program,
light metrics program, and no metrics program were the
different positions posted.
Initially the twenty four stakeholders in the decision making
group were provided with the decision making issue and the
relevant positions. The decision making issue and the positions
were posted in the argumentation tree for stakeholders to
participate. Stakeholders exchanged arguments over different
positions for one week of time. After the argumentation tree
was constructed, the developed framework was applied on the
argumentation tree. Both the individual perspective and the
collective perspective sub-frameworks were applied on the tree.
This framework has then produced the results which are
explained in detail in the following sub-section.
Results –Table 1 presents the ranked list of stakeholder
opinions based on both the individual method and the collective
perspectives. S18 is the opinion of stakeholder number
eighteen. Stakeholder S18 ranks one in the outlier ranked list,
while stakeholder S7 is ranked last in the list. Stakeholder
S18’s opinion is farthest from the mean opinion of the group
while stakeholder S7’s opinion is closest to the mean opinion of
the group. From the collective perspective framework, S20’s
opinion is ranked one (outlier) and stakeholder S2’s opinion is
ranked last in the outlier list. A decision maker can also
generate the top-K list of stakeholders with outlier opinion and
S20, S8, S23, S17, S22, S19,
S12, S21, S13, S24, S5, S10,
S9, S3, S4, S16, S11, S6,
S14, S1, S18, S7, S15, S2
The results presented in Table 1 will fall in one of the cases
presented above. This classification provides a better
understanding of the opinions in a decision making group. This
is one way of understanding the dynamics involved in a
decision making group during an argumentation process. We
do not claim that a stakeholder with outlier opinion is good or
bad. Our model just identifies the outlier opinions and lets the
group decide on how to use this information.
6. CONCLUSIONS
It is important to consider every opinion in the argumentation
process during decision making. Outlier opinions do exist in
argumentation in a collaborative environment. Outlier opinions
are the one which are farther away from the mean opinion of
the group. When outlier opinions are contrasting with the rest
of the group, this could lead to more argumentation and debate.
It is also important to identify outlier opinions from both an
individual’s perspective and a group’s perspective. We also
learn from the experiment that there is a need to evaluate the
method by using several other well established outlier detection
techniques.
7. ACKNOWLEDGMENTS
We would like to thank the Intelligent Systems Center at
Missouri S&T for their support.
8. REFERENCES
[1] Araucaria,
URL:
http://araucaria.computing.dundee.ac.uk/doku.php#downl
oad, accessed on 2/15/2013.
[2] Arvapally, R.S., Liu, X. F., and Jiang, W., 2012,
“Identification of faction groups and leaders in Web-based
intelligent argumentation system for collaborative
decision support,” Proceedings of the International
Conference on Collaboration technologies and Systems
(CTS 12), pp.509 – 516.
5
[3] Bermingham, A., 2009, “Combining Social Network
Analysis and Sentiment Analysis to Explore the Potential
for Online Radicalisation,” Social Network Analysis and
Mining (ASONAM 2009), pp. 231 – 236.
[4] Breunig, M. M., Kriegel, H. P., Ng, R. T. and Sander, J.,
1999, “OPTICS-OF: Identifying local outliers,”
Proceedings of the third European conference on
principles of data mining and knowledge discovery,
Prague, pp. 262-270.
[5] ConviceMe
Start
a
debate,
URL:
http://www.convinceme.net/, accessed on 2/15/2013.
[6] Debatebase, URL: http://idebate.org/debatabase, accessed
on 2/15/2013.
[7] Gogoi, P., Bhattacharyya, D. K., Borah, B., and Kalita, J.
K., 2011, “A Survey of Outlier Detection Methods in
Network Anomaly Identification,” The Computer Journal,
Vol. 54 (4), pp. 570-588.
[8] Hawkins, D., 1980, “Identification of Outliers,” Chapman
and Hall, London.
[9] Karacapilidis, N., and Papadias, D., 2001, “Computer
supported argumentation and collaborative decision
making: the HERMES system,” Information Systems,
Vol. 26, pp. 259–277.
[10] Karacapilidis, N., 2006, ”CoPe_it!: Towards augmenting
collaboration and learning in communities of practice,”
Workshop in Modelling Meetings, Argumentation and
Discourse, in Proceedings of the first International
Conference on Computational Models of Argument.
[11] Karacapilidis, N., Tzagarakis, M., Karousos, N., Gkotsis,
G., Kallistros, V., Christodoulou, S., and Mettouris, C.,
2009,” Tackling cognitively-complex collaboration with
CoPe_it!,” International Journal of Web-based learning
and teaching technologies, Vol. 4(3), pp. 22-38.
[12] Klein, M., 2010, “Using Metrics to enable large-scale
deliberation,” Collective Intelligence in organizations: A
workshop of the ACM Group 2010 Conference, Sanibel
Island, Florida.
[13] Klein, M., 2012a,”How to harvest collective wisdom on
complex problems: An introduction to the MIT
Deliberatorium,” MIT Center for Collective Intelligence,
URL:
http://cci.mit.edu/klein/papers/deliberatoriumintro.pdf, Date accessed: 2/14/2013.
[14] Klein, M., 2012b, “Enabling Large-Scale Deliberation
using attention-mediation metrics,” Journal of computer
supported cooperative work (CSCW), Vol. 21, pp. 449473.
[15] Kriegel, H. P., Schubert, M., Zimek, A., 2008, “AngleBased Outlier Detection in High-dimensional Data,”
Proceedings of the 14th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining
(KDD '08), Las Vegas, NV.
[16] Liu, X. F., and Raorane, S., and Leu, M., 2007,”A Webbased Intelligent Collaborative System for Engineering
Design,” in Li. W. D., Ong, S. K., Andrew, Y. C., and
McMahon, C., Collaborative product design and
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
6
manufacturing methodologies and applications, Springer,
pp. 37-58.
Liu, X. F., Khudkhudia, E., Wen, L., Sajja, V., and Leu,
M., 2009,”An Intelligent Computational Argumentation
System for Supporting Collaborative Software
Development Decision Making,” in Meziane, F., and
Vadera, S. Artificial Intelligence Applications for
Improved Software Engineering Development, IGI
Global, pp. 167-180.
Liu, X. F., Wanchoo, R., and Arvapally, R. S.,
2011,”Empirical study of an intelligent argumentation
system in MCDM,” Proceedings of the 2011 IEEE
International Symposium on Collaborative Technologies
and Systems (CTS 2011), Philadelphia, pp. 125-133.
Ng, R. T. and Sander, J., 2000,”LOF: Identifying densitybased local outliers,” Proceedings of the 2000 ACM
SIGMOD international conference on management of
data, Dallas, pp. 93-104.
Paolucci, M., Suthers, D., and Weiner, A.,
1995,”Belvedere:
Stimulating
Student’s
Critical
Discussion,” Proceedings of the Human Computer
Interaction, Denver CO, pp. 123-124.
Ramaswamy, S., Rastogi, R., and Shim, K.,
2000,”Efficient algorithms for mining outliers from large
data sets,” Proceedings of the SIGMOD international
conference on Management of data, pp. 427-438.
Satyavolu, M., 2010,”Contribution-based priority
assessment in a web-based intelligent argumentation
network for collaborative software development,” Thesis,
Missouri University of Science and Technology, Rolla.
URL:http://scholarsmine.mst.edu/thesis/pdf/Satyavolu_09
007dcc8082f337.pdf , February 15th, 2013.
Sillince, J. A. A., 1995,”A model of social, emotional and
symbolic aspects of computer-mediated communication
within organizations,” Journal of Computer Supported
Cooperative Work (CSCW), Vol. 4 (1), pp. 1 – 31.
Sillince, J. A. A., and Saeedi, M. H., 1999,”Computermediated communication: problems and potentials of
argumentation support systems,” Journal of Decision
support systems (DSS), Elsevier, 26, pp. 287 – 306.
Tang, J., Chen, Z., Fu, A. W., and Cheung, D., 2000,”A
robust outlier detection scheme for large data sets,” Sixth
international pacific – Asia conference on knowledge
discovery and data mining, pp. 6-8.
Toulmin, S., 1958,” The uses of argument,” Cambridge,
Cambridge University Press.
Turkett, W. H., Rose, J. R., and Huhns, M. N.,
2003,”Massive Deliberation,” IEEE Internet Computing.
Vesic, S., Ianchuk, M., and Rubtsov, A., 2012, “The
Synergy: A platform for argumentation-based group
decision making,” Computational Models of Argument,
Berheij, B. eds., IOS Press, pp. 501-502.
Wiebe, J., Breck, E., Buckley, C., Cardie, C., Davis, P.,
Fraser, B., Litman, D., Pierce, D., Riloff, E., Wilson, T.,
Day, D., and Maybury, M., 2003,”Recognizing and
Organizing Opinions Expressed in the World Press,”
Working Notes – New Directions in Question Answering,
AAAI Spring Symposium Series, pp. 24 – 26.
[30] Gordon, T., and Walton, D., 2006, “The Carneades
Argumentation Framework Using Presumptions and
Exceptions to Model Critical Questions,” Proceedings of
the 2006 conference on Computational Models of
Argument, pp. 195-207.
7