CRPIT Volume 106 - User Interfaces 2010
Visualisation Tools for Exploring the Uncertainty-Risk Relationship in
the Decision-Making Process: A Preliminary Empirical Evaluation
Mohammad Daradkeh, Alan McKinnon, and Clare Churcher
PO Box 84 Lincoln University
Lincoln 7647 Canterbury
mohammad.daradkeh@lincolnuni.ac.nz, {mckinnon, churcher}@lincoln.ac.nz
Abstract
Models play an important part in the decision-making
process. However, due to uncertainty in a model’s input
variables, making decisions involves a degree of risk. We
have developed two visualization prototypes for exploring
the influence of uncertainty in the values of the input
variables on the risk associated with the decision-making.
The first prototype is the interactive tornado diagram,
which is considered as an extension to the static tornado
diagram. The second prototype is the Uncertainty
Influence Explorer (UIExplorer). This paper presents and
discusses the results of an experiment conducted to assess
the efficacy of these prototypes and compare their ability
to help people answer meaningful questions related to the
risk associated with decision-making. The results show
that participants using UIExplorer performed better in
terms of accuracy and time taken to complete the
questions. Also, they found it easier to use and had higher
confidence in the decisions being made.
Keywords: Information visualization, Interaction Design,
Decision-making process, Uncertainty, Risk, Sensitivity
analysis.
1
Introduction
All decisions are intended to bring about some future
benefit to someone or something, and involve choices
(e.g. whether to buy a new machine, whether to
implement design A or B, etc.) (Willows & Connell,
2003). Systematic approaches to decision-making usually
involve models which give a quantitative estimate of the
value for the decision-maker to base a decision on
(Clemen & Reilly, 2001). For example, financial
managers use net present value and internal rate of return
for analysing investment alternatives (Dayananda,
Harrison, Herbohn, & Rowland, 2002; Jovanovic, 1999).
In another decision context such as water management,
decision-makers use more complex models to rank
multiple water management options or compare the
frequency and extent of various flooding events (Hyde,
Maier, & Colby, 2005; Xu & Tung, 2008; Xu, Tung, Li,
& Niu, 2009).
Copyright (c) 2010, Australian Computer Society, Inc. This
paper appeared at the 11th Australasian User Interface
Conference (AUIC 2010), Brisbane, Australia, January 2010.
Conferences in Research and Practice in Information
Technology, Vol. 106. P. Calder, C. Lutteroth, Eds.
Reproduction for academic, not-for profit purposes permitted
provided this text is included.
42
There are different kinds of decisions (Harris, 1998);
some decisions such as which company’s shares to buy,
involve making a choice among alternatives while others
such as whether or not to invest in a new business are
more “yes/no” decisions. Whatever the type of decision,
the decision-maker can never be certain of the values of
the variables or parameters used in the model and there
may also be errors or approximations in the model itself
(Jovanovic, 1999). For this reason, descriptions of the
decision-making process include a sensitivity analysis
step once the “best” decision has been identified (Clemen
& Reilly, 2001; Larichev & Moshkovich, 1995; Roy &
Vincke, 1981). This is evident in Figure 1 which shows a
typical decision-making process (Clemen & Reilly, 2001).
Figure 1: A Decision Analysis Process flowchart
(Clemen & Reilly, 2001).
Many researchers have emphasized the role of
sensitivity analysis in decision-making (Clemen & Reilly,
2001; French, 1986; Triantaphyllou, 2000). After
choosing the “best” alternative and following the
completion of the decision analysis, sensitivity analysis
should be carried out to investigate how uncertainty in the
input variables and criteria weights (preferences) affects
the values of decision criteria, as well as the final ranking
Proc. 11th Australasian User Interface Conference (AUIC2010), Brisbane, Australia
of alternatives (Clemen & Reilly, 2001). Several
sensitivity analysis methods have been proposed (Guillen,
Trejos, & Canales, 1998; Hutton & Charles, 1988;
Mareschal, 1988; Ringuest, 1997; Soofi, 1990;
Triantaphyllou & Sanchez, 1997). These methods are
carried out to investigate the relationship between
changes in the criteria weights and the subsequent
alteration that occurs in the ranking of alternatives (Hyde
et al., 2005). Despite this, decision-making processes are
often applied with little consideration given to uncertainty
in the input variables and propagation of such
uncertainties through the decision model (Basson &
Petrie, 2007; Xu & Tung, 2008). We suggest this is
because the way in which the sensitivity analysis step
should be used as part of the overall decision-making is
not at all clear.
We are investigating the use of visualization tools to
enable the consideration of the uncertainty to be an
integral part of making the decision rather than treating it
as an add-on step which does not have a clear role in the
process. To date we have applied these ideas to "yes/no"
decisions. Application to decisions involving multiple
alternatives will be the subject of later work.
We have proposed three visualization tools to allow
the consideration of uncertainty to be better integrated
into the decision-making process for “yes/no” decisions.
The first visualization we investigate is the well known
static tornado diagram (Cooke & Van Noordwijk, 2000;
Koller, 2005, 2007) which is commonly used to explore
sensitivity in financial decision-making. For the second
visualization we have added interactivity to the tornado
diagram. The third visualization is a prototype of our
Uncertainty Influence Explorer (UIExplorer). It has been
developed to explicitly allow the decision-maker to
explore how the risk of making an undesirable decision is
affected by the uncertainty in the input variables as
depicted by the process shown in Figure 2. The decisionmaker specifies the risk criterion to be used (e.g. that the
NPV is less than zero) and also the uncertainty range for
each input parameter.
Risk criterion
Specify
Risk calculator
Input uncertainties
Risk
Model
Decision-maker
Decision
Specify
Figure 2: Uncertainty-Risk relationship in the decisionmaking process.
This paper describes an empirical evaluation and
presents its results. We introduce an experimental design
with two parts: a quantitative part that measures and
compares the performance of participants during the
experiment. Two variables of performance (accuracy of
the answers and time to complete questions) for each
visualization were measured. In the qualitative part, the
participants answered a number of questions regarding
their experience and satisfaction with these visualizations
in the decision-making process.
This paper is structured as follows. Section 2 briefly
describes the visualizations using a widely used financial
model for investment decisions called Net Present Value
(Dayananda et al., 2002; Jovanovic, 1999). Section 3
describes the evaluation. Section 4 presents and discusses
the experiment results. Future work and conclusions are
discussed in Section 5.
2
Description of the Visualization Tools
In this section, we describe the visualizations using a
widely used financial model for investment decisions
called Net Present Value (Dayananda et al., 2002;
Jovanovic, 1999). NPV is a financial model used in
capital budgeting to analyse the profitability of an
investment or project. A positive NPV indicates that the
investment is acceptable. If NPV is negative, the
investment should properly be rejected. However, there
are many estimations and subtle interactions between
variables that have significant effects on the profit
outcomes. A basic version of calculating NPV is given by
equation 1:
(1)
Where
t is the time of the cash flow.
N is the total time of the project.
r is the discount rate (the rate of return that could be
earned on an investment.)
Ct is the net cash flow (inflow minus outflow) at time t.
2.1
Static Tornado Diagram
A tornado diagram is a pictorial representation of the
contribution of each input variable to the output of the
decision making model (Clemen & Reilly, 2001; Cooke &
Van Noordwijk, 2000; Koller, 2005). It consists of
stacked horizontal bars, each one associated with one
input variable. Each horizontal bar represents the range of
the output (NPV) as the corresponding variable is varied
over its specified range while all other variables remain
constant at their nominal values. The length of the bar
indicates the variable's effect on the model output. The
model output has a nominal value which is calculated for
the nominal values of all the input variables and displayed
as a vertical line on the diagram.
A typical diagram is shown in Figure 3. The left and
right bar ends indicate the corresponding upper and lower
bounds of NPV as the related variable is varied within its
specified range while the other variables remain constant
at their nominal values. The example in Figure 3 shows
that, for the given values of the other parameters, the
NPV is mostly influenced by varying the inflow, while the
variation in the outflow has little effect on NPV. One of
the main drawbacks of the tornado diagram is that it
assumes all of the input variables are independent. Thus,
it ignores the influence of the interaction between input
variables that might have a significant effect on the output
(Koller, 2005). In addition, it is a static representation of
43
CRPIT Volume 106 - User Interfaces 2010
the sensitivity and thus it doesn’t allow users to
interactively explore and compare possible outputs under
different scenarios.
Figure 3: The static tornado diagram shows the
sensitivity of NPV to the variation in each input
variable while other variables are held constant
(Source: own Figure).
2.2
Interactive Tornado Diagram
This visualization prototype is an extension of the static
tornado diagram where users can change the nominal
values of the input variables interactively and so explore
the effects of the interaction between input variables on
the model’s output. As shown in Figure 4, this can be
investigated by varying an input variable (with its scroll
bar) and observing how the bars on the diagram for the
other input variables change. For example, if the user
scrolls the inflow scroll bar, he/she will notice that the
length of the rate bar will change. This means that the
uncertainty in the model’s output resulting from the
variation in the rate variable is affected by the inflow
value, as can be seen from Figures 4b and 4c. In this way,
the prototype can help in exploring the influence of the
interaction between input variables on the overall output
uncertainty.
One drawback with the interactive tornado diagram is
due to how the displayed information is calculated. The
calculation used to display the horizontal bars depends on
varying one input variable at a time while keeping the
other inputs at constant values. The problem with this way
of calculation is that, in most cases, the true influence on
the output variable depends not only on the values of the
input variables, but also on the interaction between input
variables over all ranges. Thus, varying one input variable
while keeping the others constant might not, reveal the
true influence of any input variable. This led to the
development of another prototype that rectifies this
drawback.
2.3
Uncertainty Influence Explorer
(UIExplorer)
The aim of this prototype is to visualize the risk due to
uncertainty in the model’s input variables, and its
sensitivity to the variation in these input variables. The
UIExplorer uses colours to convey the risk magnitude (in
44
this case the probability of making a loss). Figure 5 shows
a screenshot of the prototype. Yellow means no risk (i.e.,
probability of making a loss =0) and dark orange
represents the highest risk (i.e. probability of making a
loss =1). The colour of a cell shows the risk associated
with that value of the input variable. It is calculated by
taking every possible value of all the other variables and
calculating what proportion of these combinations will
result in a loss. The user can retrieve the numerical value
of the risk (i.e., the probability of a negative NPV) by
pointing to any of the cells. For example if the inflow is
$30000 (highlighted cell in Figure 5) then if we consider
all other possible combinations of values for the other
variables about 44% will result in a loss (probability
0.44).
By looking at the displayed range of colours that
represent the risk magnitude, the user can quickly and
easily see where the investment is potentially risky or
where it is not. In addition, he/she can readily see the
values of the input variables for which the investment will
likely be profitable (i.e., low risk). For example, in Figure
5, if the inflow varies within the range [$33000, $35000],
the user can have confidence that the NPV will be
positive (i.e., there is no risk of making a loss if the other
variables stay within the given ranges).
Clicking on a cell in the first grid fixes the value of
that input variable. The chosen cell is highlighted, and the
colours in the lower grid change to convey a new range of
risks associated with the values of the other input
variables. For example, the lower grid in Figure 6 shows
the risk associated with the values of inflow, outflow, and
rate based on fixing the Initial Investment at $90000 and
allowing the other input variables to vary within their
ranges. The new range of colours in the lower grid
represents the recalculated risk which is calculated by
fixing the initial investment at $90000 and taking every
possible value of the other variables and calculating what
proportion of these combinations will result in a loss.
This is useful when the user wants to fix a value for a
certain variable and explore the risk range associated with
the other input variables.
The range of input variables can be modified by
changing their minimum and maximum values and as a
result, the range of colours in both grids will change to
convey the new values of risk. For example, Figure 7
shows the effect of changing the range of Inflow, from
($25000, $35000) to ($20000, $30000). It can be seen
from Figure 7 that the colours in the upper grid are darker
which means that the risk is very high for almost all of the
values of the input variables. However, this does not
mean that the risk associated with all possible scenarios is
very high. For example, when the user chooses the value
$30000 from inflow (the highlighted cell), he/she can
notice from the lower grid that there are some
combinations of the other variables which have low or no
risk. This is helpful when the decision maker wants to
assess the risk of choosing different scenarios and allows
him/her to answer many different “what-if” questions.
Proc. 11th Australasian User Interface Conference (AUIC2010), Brisbane, Australia
a
b
c
Figure 4: (a) Screenshot of interactive tornado diagram. (b) The influence of decreasing the Inflow value on the
Rate bar length. (c) The influence of increasing the Inflow value on the Rate bar length.
Figure 5: A screenshot of the UIExplorer showing the ranges of risk.
45
CRPIT Volume 106 - User Interfaces 2010
Figure 6: A new range of colours in the second grid after holding the Init. Investment at $90000.
Figure 7: The influence of changing the Inflow Range on the calculated risk and the colour range.
In this version of UIExpleorer, we display the
information in a uniform grid. The use of a grid layout
facilitates the presentation of uncertainty and the
associated risk in an organised way. In addition, it makes
it easier to see and follow the change in the risk
magnitude across the rows, which in turn facilitates the
discovery of trends and relationships between uncertainty
and risk. In addition, all input variables are bounded by a
known maximum and minimum and, for this case study,
we have made the assumption that all values in between
occur with equal likelihood. Therefore, they can be
mapped onto equal-sized cells. This way, the user can run
through or compare several scenarios with various values
and easily determine the risk associated with each value
or scenario.
In this version of UIExplorer there is a limit to the
number of divisions. We have divided each input variable
into eleven divisions although there is no specific reason
for that number. However, for finer-grained analysis and
representation the user can change the min and max
values. This facilitates more detailed assessment over a
small range, and consequently more precise and effective
decision making. In the future versions we plan to make
the UIExplorer more flexible and amenable to larger
number of variables.
Colour was chosen for the purpose of presenting risk
because it is widely used for risk visualisation and
communication (Bostrom, Anselin, & Farris, 2008). Also
46
it is an important visual attention guide that can highlight
levels of risk (Wolfe & Horowitz, 2004). Several studies
have addressed the influence of colours on risk perception
and decision-making processes (Lipkus & Hollands,
1999; Rogers & Groop, 1981; Soldat & Sinclair, 2001;
Wogalter, Conzola, & Smith-Jackson, 2002). A study by
Wogalter et al. (Wogalter et al., 2002) supports the
following hierarchy of colours to convey risk level:
red/dark orange riskier than yellow, yellow riskier than
green. Scaling based on lightness or brightness may be
helpful in presenting risk information (Bostrom et al.,
2008). Davis and Keller (Davis & Keller, 1997) asserted
that using colour hue and colour value are the “best
candidates” for presenting risk information using static
methods. Brewer (Brewer, 2006) advises use of light-todark colour for low-to high values with a constant hue.
3
Evaluation
We evaluated the visualizations by testing and observing
how people used them in a controlled experiment. The
aim was to determine whether answering questions using
the three visualizations would differ with respect to
accuracy, completion time, ease of use, and confidence in
the decisions that have been made. Specifically, we try to
address the following questions:
1. How easily and quickly can users assess the
magnitude of risk of making a loss associated
with a certain value or a range of values?
Proc. 11th Australasian User Interface Conference (AUIC2010), Brisbane, Australia
2.
How easily and successfully can users answer a
variety of “what-if” questions?
3. Can users successfully use the interactive options
to assist the risk assessment process?
4. How confident are users in the decisions they
make using these tools?
In the following sections, the method of the evaluation
will be explained, and then the results will be discussed.
3.1
Experimental Design
evaluate the visualizations and would have little trouble
understanding the scenario.
Average profile of participants (1:None, 2:Beginner, 3:Intermediate, 4:Advanced)
Risk Assessment
2.3
Decision Making under
uncertainty
2.3
Financial Modeling
(e.g. Net Present Value)
2.6
11:None
2:Beginner
2
33:Intermediate
4:Advanced
4
For this experiment, the Net Present Value (NPV) model
is used. The four input variables are Initial investment,
cash inflow, cash outflow, and discount rate. The output is
the net present value, calculated over a number of years.
We put the participant in the situation of deciding whether
or not to make an investment as follows:
“You are planning to make an investment and you
need to make a decision based on the value of the NPV.
You are uncertain about the exact values of the model’s
input variables so there is a risk involved in your decision.
You need to deal with this uncertainty and assess the risk
of your decision. The risk here means the probability of
making a loss.”
The participants were given a brief introduction to
each visualization and the experimental method and were
then asked to answer a set of questions using each
visualization. During this time they were observed and
their performance was recorded. Upon completion of the
questions, the participants were asked to complete an exit
questionnaire. Participants took part in the experiment
individually with an observer present to record the time
taken to answer the questions.
3.2
3.4
Participants
We recruited 10 participants from the Lincoln University
community. All participants had some understanding of
the use of financial models and information to analyse and
interpret data relating to business activities. Five were
undergraduate students; all of them have enrolled in the
Financial Information for Business Paper (ACCT103).
One MSc student, with research experience in developing
models for financial forecasting using neural networks,
also participated. The other four participants were PhD
students in the Faculty of Commerce studying marketing,
business management, finance, and accounting
respectively. The latter participant was also working as a
part time lecturer in accounting. Of the sample, 6 were
male and 4 were female ranging in age from late teens to
40+. All of them agreed to spend 30 minutes with our
experiment and receive a $20 voucher in compensation.
The motivation behind recruiting students from Faculty of
Commerce is their good knowledge and understanding of
the basic NPV model.
3.3
Procedure
The method used in the experiment was as follows: The
participants filled out an entrance questionnaire to
determine their background experience. It asked the
participants to rate their experience in three areas on a
four-point scale (1-None, 2-Beginner, 3-Intermediate, 4Advanced). Figure 8 shows the average rating of
participants’ familiarity with financial modelling, decision
making under uncertainty and risk assessment.
Participants rated their familiarity with financial
modelling between none and intermediate, but rated their
familiarity with decision making under uncertainty and
risk assessment between beginner and intermediate level.
This suggests that the participants were well placed to
Figure 8: Background of participants.
Test Questions
The task of the participants in the experiment was to
answer questions about the impact of input uncertainty on
the risk associated with making a decision for the given
scenario. The questions addressed tasks common to the
decision making process and required finding facts and
information to answer them correctly. The five questions
below were repeated for each visualization:
Q1. For the displayed ranges, which variable do you
think has the most effect on the risk of making a loss?
(Purpose: to find the most influential variable on risk)
Q2. Approximately, for what range of cash inflow can
you be assured that the NPV will stay > 0 i.e. there is no
risk of making a loss? (Purpose: to find a range of values
associated with a specified risk)
Q3. What do you think the risk of making a loss will
be if the Discount rate becomes 10%? (Purpose: to
determine the risk associated with a specified input value)
Q4. Given that Initial investment is fixed at $90000,
approximately what is the minimum Cash Inflow that will
ensure a positive NPV? i.e. probability of making a loss is
zero. (Purpose: to find a value within a particular
scenario)
Q5. If the Inflow is $30000, what is the range of rate
values that will ensure no risk of making a loss? (Purpose:
to determine a range of values resulting in a specified
outcome).
In addition to the total time to complete questions
using each visualization and the accuracy of answers
overall, we looked at each question individually. Each
participant was asked to rate the ease of use of each
visualization to answer each question on a four-point
scale (1-Very difficult, 2-Difficult, 3-Easy, 4-Very easy).
Participants were also asked to explain why they found a
47
CRPIT Volume 106 - User Interfaces 2010
question easy or difficult to answer. After completing the
questions for each visualization, participants were
required to rate their confidence in the decisions they had
made with these visualizations on a 5 point scale (where 1
meant Not Confident and 5 Highly Confident).
4
Evaluation Results
The results revealed considerable differences between the
visualizations.
4.1
Accuracy
In terms of the accuracy of answers, the results showed
that there is a noticeable difference between the three
visualizations. Figure 9 shows the number of accurate
answers for each question using each visualization. 7 out
of 10 participants answered the question on finding the
most influential variable on risk (Q1) correctly using the
three visualizations. This is because this question doesn’t
need interaction from participants to find the answer. For
the questions on finding a value or range of values (Q2
and Q4), the static tornado diagram failed to give very
accurate answers. This is because these questions could
not be answered without interaction. The number of
correct answers for Q2 and Q4 increased for the
interactive tornado diagram. However, moving the sliders
did not give high accuracy (5 out of 10 were incorrect for
both Q2 and Q4). Using UIExplorer, participants gave
more accurate answers for Q2 and Q4 than using static
tornado or interactive tornado diagram.
For the questions on determining the risk associated
with a certain value (Q3) or the risk associated with a
range of values within a scenario (Q5), the results show
that participants did better using UIExplorer than using
both the static tornado and the interactive tornado. This
suggests that both interactive tornado and UIExplore
helped in finding precise answers to the questions that are
related to finding the risk (Q3 and Q5). On the other
hand, interactive tornado failed to give high accuracy for
the questions that are related to finding the values that
affect the risk (Q2 and Q4), while UIExplore succeed to
give high accuracy for the same questions, as shown in
Figure 9.
Accurate answers
Number of accurate answers
12
10
10
9
9
9
9
8
8
7 7 7
Static
Tornado
6
5
5
4
Interactive
Tornado
4
4
UIExplorer
2
2
1
0
Q1
Q2
Q3
Questions
Q4
Q5
Figure 9: Comparison between the three visualizations
in terms of accuracy.
48
4.2
Time Taken
Participants took from 5 to 10 minutes to complete the
five questions using each visualization. Figure 10 shows
that the mean time taken to answer the questions was 7.3
minutes using static tornado, 6.2 minutes using interactive
tornado, and 6 minutes using UIExplorer. The mean time
was shorter for UIExplorer than interactive tornado
perhaps because, as some participants expressed, moving
the sliders in interactive tornado prototype to find the
answer takes longer than observing the colour change in
UIExplorer. Table 1 summarizes the results of the time
taken by the participants to answer the questions using
each visualization.
Time Taken on Average
UIExplorer
6
Interactive tornado
6.2
Static tornado
7.4
0
1
2
3
4
5
6
7
8
9
10
Time in minutes
Figure 10: The average time taken by participants to
complete the questions using each visualization.
visualization
Static tornado
average
sd
min
max
7.4
Interactive tornado
6.2
2.1187
5
10
1.3166
5
UIExplorer
9
6
1.1547
5
8
Table 1: Summary of the time taken Results.
4.3
Ease of Use
Figure 11 shows that there is a clear distinction between
the three visualizations in terms of their ease of use.
While it was difficult to find answers using static tornado,
it becomes easier using interactive tornado and
UIExplorer. For example, using the static tornado
prototype, participants found it difficult to obtain answers
to the questions related to finding the values of the input
variables (Q2 and Q4) and the risk associated with these
values (Q3 and Q5). However, using the interactive
tornado prototype, participants found that the search for
the answers became easier and even much easier when
using UIExplorer.
Using the static tornado prototype, participants found
that the question related to finding the most influential
input variable on the risk (Q1) easier than the other
questions. On the other hand, although the ease of use for
the interactive tornado prototype and UIExplorer was
rated between very easy and difficult, users found
UIExplorer much easier than the interactive tornado. This
was consistent with the feedback from the participants,
who mostly expressed their satisfaction with the use of
UIExplorer.
Proc. 11th Australasian User Interface Conference (AUIC2010), Brisbane, Australia
On average, participants found UIExplorer easier to
use than the static tornado and interactive tornado as
shown in Figure 12. Table 2 summarizes the results of the
participants’ evaluation to the ease of use. It is noted here
that two of the participants who answered Q2 and three
participants who answered Q4 incorrectly assessed
finding the answers as easy. This gives the impression that
the interactive tornado prototype is misleading because
some of the participants have been unable to identify the
correct answers despite what they perceived as a
reasonable ease of use.
associated with each value can be directly seen without
the need for further interpretation. It should be noted that
although some of the participants responded with
incorrect answers, their level of confidence using the
interactive tornado prototype was high. This again
indicates that the interactive tornado prototype leads to
misleading interpretations concerning the risk associated
with the decision making.
Visualization
Static Tornado
Interactive Tornado
UIExplorer
The ease of use rating for each question in each visualization
3.5
3.5
Average
min
1
2
2
max
3
5
5
2.9
3.3
3.1
3.2 3.3
3.3
3.2
3
2.8
2.5
3.5
4.5
2.3
2.1
1.9
2
Static
tornado
1.9
1.5
Interactive
tornado
1
UIExplorer
0.5
0
Q1
Q2
Q3
Q4
Q5
Questions
Figure 11: Comparison between the ease of use for each
question in each visualization.
Easy of Use on average
3.38
UIExplorer
Interactive tornado
3.06
2.22
1
1.5
2
2.5
3
3.5
1: Very diffecult
4
4:Very easy
Figure 12: A comparison between the three
visualizations in terms of ease of use.
Visualization
Static tornado
Interactive tornado
UIExplorer
average
sd
min
max
2.22
3.06
3.38
0.6788
0.5858
0.6023
1
2
2
4
4
4
Table 2: Summary of the ease of use results.
Level of Confidence
Results summarized in Table 3 show that there is a clear
difference in the confidence of the participants for the
decisions that can be made based on the three
visualizations. Participants showed a low level of
confidence in their decisions using the static tornado
prototype (2). This level rose noticeably for the
interactive tornado prototype (3.7) and rose to (4.3) for
UIExplorer. One of the main features of UIExplorer, as
most of the participants expressed, is that the risk
Post-study Questionnaire
Eight of the participants found UIExplorer easier to use,
more intuitive, and quicker to understand and learn. They
understood the representation of risk by colours. On the
other hand, two participants preferred the interactive
tornado diagram because they found that moving the
sliders to reach the required values was intuitive and lead
to more understanding and easier assessment. This is
supported by their correct answers when using the
interactive tornado prototype.
We didn’t notice that participants faced a problem in
distinguishing colours/gradients. Participants were able to
relate between the degree of colour and the degree of risk.
They were also able to understand the relationship
between uncertainty and risk through gradations of
colours. This is consistent with the results of the
experiment which showed that the overall performance of
participants was better using UIExplorer.
5
Static tornado
4.4
sd
0.8165
0.82327
0.94868
Table 3: confidence level results.
4
3
average
2
3.7
4.3
Conclusion and future work
This paper presents the results of an experiment
conducted to compare the ability of three visualization
prototypes to help people explore the influence of
uncertainty in the input variables on the risk associated
with the decision making. The first visualization is a static
tornado diagram. The second visualization is an
interactive tornado diagram, which is a modification of
the static tornado diagram. The third visualization is the
uncertainty influence explorer.
The results show that most of the participants preferred
to use UIExplorer rather than the static or interactive
tornado prototypes. The use of UIExplorer leads to more
correct answers and a shorter time taken to find those
answers. Participants found UIExplorer easier to use and
they had greater confidence in their decisions compared
with the static tornado and interactive tornado prototypes.
This was consistent with the quantitative results.
Participants’ feedback confirmed that further research is
needed to improve the design of the UIExplorer so that
the user can explore the risk associated with the decision
making models at several levels of detail. It is important
for the decision maker to be able to explore the risk
associated with each value of each input variable, the risk
associated with each input variable regardless of the
49
CRPIT Volume 106 - User Interfaces 2010
current value of that variable, and the risk associated with
each scenario.
Although there were only 10 participants in this study
and we did not completely get rid of learning effects and
bias in the results, we believe that the results clearly
indicate that our approach of including the risk of making
an acceptable decision as an integral part of the decisionmaking process has significant merit. However, further
evaluation is needed. More extensive user evaluation
would include more participants and more tasks. In
addition, we intend to develop UIExplorer further and
also explore ways of extending the approach to decisions
where there are a number of alternative options.
Acknowledgements
We would like to thank all participants without whom the
study would not have been possible.
6
References
Basson, L., & Petrie, J. G. (2007). An integrated approach
for the consideration of uncertainty in decision making
supported by Life Cycle Assessment. Environmental
Modelling & Software, 22(2), 167-176.
Bostrom, A., Anselin, L., & Farris, J. (2008). Visualizing
Seismic Risk and Uncertainty. Annals of the New York
Academy of Sciences, 1128(Strategies for Risk
Communication Evolution, Evidence, Experience), 2940. doi:10.1196/annals.1399.005
Brewer, C. A. (2006). Basic Mapping Principles for
Visualizing Cancer Data Using Geographic Information
Systems (GIS). American Journal of Preventive
Medicine,
30(2,
Supplement
1),
S25-S36.
doi:10.1016/j.amepre.2005.09.007
Clemen, R. T., & Reilly, T. (2001). Making Hard
Decisions with DecisionTools (2nd rev. ed.). Pacific
Groce, CA: Duxbury Thomson Learning.
Cooke, R. M., & Van Noordwijk, J. M. (2000).
Generalized Graphical Methods for Uncertainty and
Sensitivity Analysis. Bashkir Ecological Journal,
(Special Issue) 1(8), 54-57
Davis, T. J., & Keller, C. P. (1997). Modelling and
visualizing multiple spatial uncertainties. Comput.
Geosci.,
23(4),
397-408.
doi:260799
http://dx.doi.org/10.1016/S0098-3004(97)00012-5
Dayananda, D., Harrison, S., Herbohn, J., & Rowland, P.
(2002). Capital Budgeting: Financial Appraisal of
Investment Projects. Cambridge: Cambridge University
Press.
French, S. (1986). Decision Theory: An Introduction to
the Mathematics of Rationality. Chichester, U.K: Ellis
Horwood.
Guillen, S. T., Trejos, M. S., & Canales, R. (1998, 8-12
June 1998.). A robustness index of binary preferences.
Paper presented at the XIVth International Conference
on Multiple Criteria Decision Making, Charlottesville,
Virginia.
Harris, R. (1998). Introduction to Decision Making.
(August
15,
2009).
Retrieved
from
http://www.virtualsalt.com/crebook5.htm
50
Hutton, B., & Charles, P. S. (1988). Sensitivity analysis
of additive multiattribute value models. Oper. Res.,
36(1),
122-127.
doi:http://dx.doi.org/10.1287/opre.36.1.122
Hyde, K. M., Maier, H. R., & Colby, C. B. (2005). A
distance-based uncertainty analysis approach to multicriteria decision analysis for water resource decision
making. Journal of Environmental Management, 77(4),
278-290.
Jovanovic, P. (1999). Application of sensitivity analysis
in investment project evaluation under uncertainty and
risk. International Journal of Project Management,
17(4), 217-222. doi:10.1016/s0263-7863(98)00035-0
Koller, G. (2005). Risk Assessment and Decision Making
in Business and Industry: A Practical Guide: CRC
Press.
Koller, G. (2007). Modern Corporate Risk Management:
A Blueprint for Positive Change and Effectiveness: J.
Ross Publishing.
Larichev, O. I., & Moshkovich, H. M. (1995). ZAPROSLM—A method and system for ordering multiattribute
alternatives. European Journal of Operational
Research 82(3)(3), 503-521.
Lipkus, I. M., & Hollands, J. G. (1999). The Visual
Communication of Risk. J Natl Cancer Inst Monogr,
1999(25), 149-163.
Mareschal, B. (1988). Weight stability intervals in
multicriteria decision aid. European Journal of
Operational Research, 33, 54–64.
Ringuest, J. L. (1997). Lp-metric sensitivity analysis for
single and multi-attribute decision analysis. European
Journal of Operational Research, 98, 563–570.
Rogers, J. E., & Groop, R. E. (1981). Regional Portrayal
With Multi-Pattern Color Dot Maps. Cartographica:
The International Journal for Geographic Information
and Geovisualization, 18(4), 51-64.
Roy, B., & Vincke, P. (1981). Multicriteria analysis:
survey and new directions. European Journal of
Operational Research, 8, 207–218.
Soldat, A. S., & Sinclair, R. C. (2001). Colors, smiles,
and frowns: External affective cues can directly affect
responses to persuasive communications in a mood-like
manner without affecting mood. . Social Cognition, 19
469–490.
Soofi, E. S. (1990). Generalized entropy-based weights
for multiattribute value models. Operations Research,
38, 362-363.
Triantaphyllou, E. (2000). Multi-criteria decision making
methods: A comparative study. Boston: Kluwer
Academic Publishers.
Triantaphyllou, E., & Sanchez, A. (1997). A sensitivity
analysis approach for some deterministic multi-criteria
decision-making methods. Decision Sciences, 28, 151–
194.
Willows, R., & Connell, R. (2003). Climate Adaptation:
Risks, Uncertainty and Decision-Making (Technical
Report). Oxford, UK: UKCIP.
Proc. 11th Australasian User Interface Conference (AUIC2010), Brisbane, Australia
Wogalter, M. S., Conzola, V. C., & Smith-Jackson, T. L.
(2002). Research-based guidelines for warning design
and evaluation. Applied Ergonomics, 33(3), 219-230.
doi:10.1016/s0003-6870(02)00009-1
Wolfe, J. M., & Horowitz, T. S. (2004). What attributes
guide the deployment of visual attention and how do
they do it? Nat Rev Neurosci, 5(6), 495-501.
Xu, Y., & Tung, Y. (2008). Decision-making in Water
Management under Uncertainty. Water Resources
Management, 22(5), 535-550.
Xu, Y., Tung, Y., Li, J., & Niu, S. (2009). Alternative risk
measure for decision-making under uncertainty in water
management. Progress in Natural Science, 19(1), 115119.
51