>> Danyel Fisher: Great. Good morning everyone. I'm happy to announce and
introduce Remco Chang. He's a visiting research scientist from UNC Charlotte working
at the Charlotte Visualization Center. He's going to be talking today about his
visualization techniques.
By the way, I've seen some comment and notes, so I should point out that the "George in
a box in a corner" is not a demo for this talk; it's just a visitor to the talk. You're welcome
to go chat with him afterwards, beforehand, or anytime else.
Also, for those of you watching remotely, I'll be happy to pass on questions that you
might have for Remco via IM.
I'm Danyel Fisher. Again, I'd like to welcome and introduce Remco Chang today. He'll
be talking about visualization.
Remco?
>> Remco Chang: Thanks, Danyel. I really appreciate the opportunity to be here. So
I'm going to be talking about using interaction in visualization. So the title is: Thinking
Interactively with Visualizations.
So I'm going to start with a little background about how the visualization community
typically thinks about interaction. So this is not a rigorous study or anything, but I
believe that most people in our community believe that interactivity is essential to doing
visualization and analysis.
So I have some quotes here. The first one is from [inaudible] which is kind of the
research agenda for the vast visual analytics community. And it says: A visual analysis
session is more of a dialogue between the analyst and the data. The manifestation of this
dialogue is the analyst's interactions with the data representation.
And, similarly, there's an InfoVis paper from two years ago that says basically without
interaction a visualization is just static images that you can't really do much with, or
animated images.
So it's pretty easy to believe that's true. But as it turns out, if you kind of go through
literature, there's really been fairly limited amount of research in our community that
specifically focuses on interaction. So what ends up happening is like what Yi and
Stasko says from Georgia Tech is that people typically try to do visualization and then
add interaction to the visualization as kind of a secondary thought.
There's really been limited amount of work where the focus is on interaction and the
visualization is maybe perhaps a second role.
So the goal of this talk is to consider the role of interaction in computer graphics,
visualization -- I have to take out the word information, but visualization in general and
visual analytics.
So the first thing I have to do is take you all the way back to kind of where I got started
thinking about this problem. And it really started about 15 years ago when I was doing
my master's thesis. And back then I was doing offline rendering of animated characters.
This is at Brown University when graphics was kind of just on the rise back then. And
this is offline rendered images. And the thesis was really about kinematic motion,
deformable meshes and all that stuff.
But the point about this is it's very computationally heavy and for a 10-second sequence it
would literally take all night. So before I go home -- I'll code all day. Before I go home I
would set a bunch of patches to run on various machines, and then I'll come in the next
day to see what the results are.
So in that sense it was actually really hard to debug what I was doing because effectively
you have to remember what you saw last time or you have to review it from last time and
you really don't have a good sense of the, you know, 50 parameters you're tweaking and
what their relationships are with each other.
So there's kind of an offline process, this lack of interaction, makes it really hard to think
about the differences between each run.
So kind of thinking about what would it take to make this process more interactive, and if
you just kind of flip through literature, people would typically say about 12 frames a
second is necessary. And what that means is rendering a frame under .08 per second.
And in the case of my thesis and among many other things, this is often really, really
difficult, because you have really complex scenes with lots of polygons or a lot of
information, at which point the computation becomes heavy. So the things you have to
do is you have to simplify the scene, you remove some detail and you have hierarchies of
detail, just to keep things interactive.
And to give you an example of what I think of simplification, I'm using my example of
urban simplification that I did a few years ago. And I'm showing you first the results. So
what we have here are three images of rendering or simplifying an urban model, and you
have the original model on the left, the middle one is about 45 percent of the polygon
count, and the last one is about 18 percent.
And the key point about this is why you see subtle differences, like, for example, the last
image looks a little bit dimmer than the first one, but the key point is that during the
simplification process you retain kind of the important features of the city.
So the question then becomes, well, how did you choose the right polygons to remove.
And this is kind of the crux of what simplification in my opinion should be about.
So here I show you some examples of simplification. In the upper left corner we have the
original urban model. That's if you're looking from the topdown view. If you simplify
this model using kind of a traditional method like QSlim, which is basically all based on
geometry, what you end up having are these kind of models where it doesn't really make
a lot of sense. So semantically it's not quite the same as the original.
Whereas our approach was to say, well, using a scene polygon count instead of
decimating it, let's aggregate them, and then we slap textures onto them.
And if you compare the original with our texture model, it looks pretty similar. But
what's really interesting about this project was that we were trying to calculate
quantitatively whether or not our approach was indeed better. And it turned out that it's
not really.
So what we did was we took these three models and we ran a flag-through scene and we
counted the number of pixels that are different from the original scene. And we plotted
this in terms of a percentage, so this will be 5 percent or so. The actual number is not
really the important part, but the important part is when you look at these three images,
three models, they look really different. But quantitatively they are similar. The three
lines are almost indistinguishable.
So there's a difference between quantitative and semantic, meaning doing the
simplification process, which traditional simplification does not really take into account.
And in this particular case when we're thinking about the city, the goal of the
simplification has to be about retaining the image of the city or the semantically
important part of the city.
And basically what we did within our application was to borrow architectural theories or
urban theories. In this particular case we use this concept called urban legibility which
basically describes a city as saying the most important aspects of a city are paths, edges,
districts, nodes and landmarks.
And this is based on Kevin Lynch's work, who's an architect at MIT in the '60s. And
really he was way ahead of his time in my opinion. What he did was he sent a bunch of
grad students out to Boston and he said we'll just go out there and drive around. And
when they came back he asked them to start sketching what they remembered.
So they came up with a stack of these kind of sketch maps and Kevin Lynch basically sat
there and flipped through this and said, well, what are the most important things that
people remember during these sketches. And this is where he compiled the five
elements.
So this is really interesting work to me, but the core point about this is about the
simplification aspect. But to take a little step back in thinking about interaction and about
the semantics of a city, what we realize is that when people think about a city, they don't
necessarily just think about the geometry.
So, for example, we did an informal survey and said, well, tell me what you think about
New York. And what we get are these kind of responses where people say New York is
large, compact, and crowded, or somebody said the area where I lived there was a strong
mix of ethnicities.
So if you take these two sentences and you kind of dissect a little bit, what you realize is
that there are geometric aspect, where you talk about large and compact; there are
information aspect of a city, where you talk about crowded and talk about ethnicities; and
lastly there's a very important point about a person's perspective about a city. So there
there's this view dependence aspect to thinking about a city.
So what we did is we wanted to create a visualization that doesn't just focus on the
geometry but focuses on the overall impression of the city.
And we end up having this information visualization application that we call UrbanVis
where -- it's a coordinated multiview where on the right-hand side you have the geometry
side, and the hierarchy of the shapes are based on the rows of legibility, and we have the
information side where we try to depict the census information. And we just picked a
few categories in terms of ethnicity, in terms of income, and so on. And we have a
matrix view and the parallel coordinates view, which I'll get into a little bit later.
But lastly we have this little yellow sphere that we have which we can allow the user to
control to change their points of interest or their view dependency, so to speak.
So I'm going to run a quick video. I usually actually demo this, but it turns out that it's
pretty hard to demo this sometimes.
So what I'm going to show you is basically how this application can be used. So, for
example, the user can drag around the little ball. And what you see is that the
information side on the left -- or on the left are updated according to what the user is
interested in.
So in this particular case, I'm showing that on top of being able to drag your point of
interest, you can change the degree of detail. So what that means is if you bring the ball
all the way up then the aggregation becomes stronger, and if you bring it all the way
down you get more detail around the sphere itself.
And in terms of interaction, what the user can do is select or highlight in any of the three
views, and the other views will correspond. So, for example, in this case we highlight a
column and you see the parallel coordinates line light up, and you see the geometry
aspect line up as well.
So two examples of how we can see if this application could be used. The first is in
comparing different parts of a city, where we're comparing downtown Charlotte to a town
about 250 miles north of downtown called Davidson, and we're trying to see how they are
different.
So what the user would do is bring the little yellow ball from Charlotte to Davidson and
we look at the matrix view. So what you see here is that the areas around where there's
red, so red is closer to where your point of focus is, you see a lot of jaggies. So basically
it's -- there are tall spikes and low spikes. So this shows a great deal of heterogeneity in
terms of ethnicity and income. Whereas if you look at the Davidson area, it's basically
flat. So this is a middle-income, white suburbia kind of a town.
So while the user is exploring, they can start looking at a matrix view to kind of get a
sense of heterogeneity around their region.
And the second scenario would be, for example, looking for ethnicity pockets around the
city, and in this case what the user does is bring the ball down to downtown Charlotte and
in the parallel coordinates view will select where there's high Hispanic population.
So what's interesting about this parallel coordinate is that while the user selects these
lines, first of all you see all the white dots line up, so it tells you where the ethnic clusters
are, but if you follow these lines alone, what you start seeing is that there are
relationships between the variables.
So in this case we see that the Hispanic population has almost a 1:1 relationship with
their citizenship status. And if you come further along, you see that there's a negative
correlation with income and housing values and so on.
So taking a step back, thinking about the interaction or interactive techniques used in
these applications, we refer to a paper again by Yi and Stasko at Georgia Tech where
they kind of categorize interactions into different uses.
So what they said is you can use interactions to select, explore, reconfigure, encode,
abstract, filter and connect, which most of these are implemented within our system and a
lot of other visualization systems as well.
But what's interesting about this is if you really kind of lump it all together, what we
believe is that what Stasko is describing is that you can use interactions to help you think.
So while you're exploring you're actually thinking about what you're looking for.
So to try to convince you, I'm going to run through some applications that we've built and
hopefully at the end of it you'll believe that interactivity can be embedded into multiple
different types of applications and it actually does help with the domain experts or the
end users.
So the first case is the WireVis project, which we did with Bank of America. And this
particular project was looking for suspicious wire transactions and we're looking for the
temporal trends as well as different suspicious activities over the course of a year.
And the way that we kind of look at the problem is to consider the transactions as
relationships between keywords, so keywords within the transactions themselves to the
accounts, or keywords to the keywords, or keywords/account over time and so on.
So, for example, keyword to keyword would be if you have an entity in China who's
selling baby food but sending money to the U.S. company who supposedly is dealing in
car parts. So the keyword "baby food" and "car part" should not really show up together,
but if they do, this is something that you might be interested in, you might want to
investigate further.
And this is kind of an overall interface. And the four different coordinated views are
used as heat map view which shows the relationship between the accounts and the
keywords. We have temporal view which looks at relationships over time. We have the
keyword network graph, which talks about when keywords are not supposed to show up
together. And then lastly we have this idea about search by example which basically says
if you find a particular pattern or account that you're interested in, show me other ones
that are kind of like it.
So I'm just going to show you this application real quick. I kind of preloaded this. So
kind of the point about this application and the interaction in it is that when you're
interacting with any variable in one view the other view will correspond.
So this in case the X axis encodes a list of the keywords. So these are predefined words
by analysts at the bank. We have to use numbers to mask it because of proprietary and
security reasons. On the Y axis we have clusters of accounts. So when I say 98 here, it
means that cluster has 98 accounts in it. And you can drill down into any of these where
eventually you'll get to individual account information.
So the bottom view of course is time. And what we can show is that when I'm interacting
with let's say the keywords, I can start to see temporal patterns.
So in this case selecting the keyword of 37, what you see is that there's this strong
temporal aspect of the keyword's occurrence. It does not show up anywhere in the first
half of the year, but it shows up more frequently on the second half.
And, further on, the network graphic can see that keyword 37 shows up along with
keyword 169, and this is what the analyst has to look at and decide if that's questionable
or if that's normal.
And, similarly, you can select a particular day, so if you highlight over a day, you can see
on that day which keywords were used and which accounts actually transact on that
keyword.
Second example I want to talk about is a project that we did with University of
Maryland's Center of Excellence on the study of terrorism and response to terrorism.
This case we're looking at the Global Terrorism Database which is compiled for the past
30 or so years. It contains about 60,000 incidents over 120 dimensions. So this is
recording all terrorist activities around the world. So it's not just domestic.
And the way that we designed the visualization was to encompass this model of the 5 Ws.
So we want to encode the who, what, where, and when, and I put why in the parentheses
because the why part, it's not really visualizable, so this is a part that you're supposed to
figure out as you're interacting.
So the interaction is designed so that the user can put in as many Ws as they know and try
to see what the correlation is.
So this is a basic overall design. We have each of these boxes correlated to the who and
the what and the where and the when. And we have this box where you can always go
back to looking at the original data, as well as this evidence box which is basically kind
of a temporary save box where if you found the particular group you're interested in,
you'll put it in there and then you can compare it to other groups later on.
So just to show you a couple examples of how this is used, here we're looking at the
activity pattern of a particular group. Actually, this is Hezbollah. And what you see is
these lines are actually connecting incidents in a temporal way. So two dots that are
connected would be temporally next to each other.
So when you look at this graph, the obvious thing that shows up is, you know, you have
this outlier where something happened in South America, turns out to be Buenos Aires.
And if you look at it a little bit further, what you realize is that Hezbollah's attacks are not
actually decided on geographic locations. So that was actually news to us because we
always thought that Hezbollah is active within the Middle East. But it turns out they're
actually everywhere. In particular, they're motivated by religious believe leafs. In the
case of what happened in Buenos Aires, it was a bombing at a mosque.
And if you look at their pattern over time, what you see is that their attack patterns
actually changed as the group developed. So in the beginning you have more
kidnapping -- sorry the font's a little bit small -- which is the green stripe. And as it went
on in time around the early '90s, they started shifting to more bombing. And then
afterwards they kind of abandoned bombing and just started doing facility attacks.
So this is kind of a temporal trend where you can see how the group's behavior is actually
changing over time.
>>: Are you missing data from '93 or ->> Remco Chang: Yes. Sorry. Missing data from '93. So that's not really us. That's
actually somehow the data collection site just had '93 disappeared forever.
So a second case is where we start with the where. So let's say we're interested in the
Philippines. And if you highlight and select a region in Philippines, what you see very
quickly is that there's a very active group in the Philippines, and this is called the New
People's Army. So it's pretty obvious it's a domestic group that have never left
Philippines, they have always been active strictly within the boundary of the country.
But, again, if you look at the temporal aspect of it, what you see is that while their attack
pattern is fairly consistent, the amount of attack actually really changed. So they kind of
took a quick rise in the '80s or 90s, and then -- sorry. The late '80s. And then in the early
'90s it just kind of disappears, and then it just completely -- completely gone forever.
So we did some research into why this happened. And by research I mean literally going
to Google, going to whatever search engine and you start searching for news articles and
see what's out there.
And it turned out that the reason why they disappeared is that in '94 they elected a new
president and one of his first act was to negotiate with all the terrorist groups and issue
pardons. So this group was pardoned and they just disappeared.
So this is the why part where you're motivated by what you see but the why part you still
have to find out.
A third example I kind of just put in really for George. This is a project that we did this
year where we're looking at the analysis of biomechanical motion.
So the premise is that biomechanical motion sequences or animation are really difficult to
analyze. And to show you why that's true, I'll show you a simple video.
This is an animation or motion sequence of a pig chewing. So this is running under half
speed. And if I were to just move it forward in normal speed, this is what the pig is
doing. And basically all you see is that pig just goes open/close, open/close, open/close.
And it's really kind of hard to see what the pig is doing. And you can watch this a million
times. You can go frame by frame. And the fact is it's really hard to see the differences
within the cycles. And as it turns out, there are differences.
So the claim that we make is that watching the movie repeatedly does not actually lead to
understanding of the sequence. And actually we didn't claim this; the biologist of -- the
evolutionary biologist we used actually said this.
So this work is in collaboration with Brown University who captured the data and people
in Minnesota, and we're looking to examine the mechanics of a pig chewing but when
they're chewing different type of foods or different amounts of food.
And one of the very obvious thing that I learned is that while you think about an animal
chewing to be just kind of open and close, as it turns out that jaw joint is very, very
complex. It can actually dislocate and just kind of do this almost like a ball joint kind of
motion. So it's not just like this, it's actually kind of like -- you know.
So when that happens, the data is basically described not just as an angle, but it's actually
described as transformation, rotation and translation of the rigid bodies. So this is kind of
the overview that we have that we designed for the evolutionary biologists. A lot of this
is motivated by George's work that I'll talk about a little bit later.
But what we wanted to do was to give the user an overview about all their data, in this
case a different color. It's a little bit hard to see. But the different color segments
described or showed different takes on the experiments. So this will represent, for
example, the pig chewing nuts or this could be chewing pig feed or whatever.
And what we want to let the user do is to select and highlight anything that they see on
the screen, and then we'll have detail view pop up where you can look at that particular
segment of the sequence more carefully. And we have additional information
visualization things to help you kind of look at your data.
But one of our key focus of this project was to look at the interactive comparison part of
looking at motion cycles. And this is definitely following George's work from last year
where he talked about how to look at animation sequences and their effects.
So some of the techniques that we used are small multiples. So this is a small-multiple
view based on the previous image. In this case what we did was we took a tracer, so we
looked at the pig's jaw and we said we'll put a trace on the bottom of the jaw and we just
let the animation play out while it traces a pattern.
So this is actually really useful, because, as you see, in a small-multiple sense, you can
see when the pig is just literally opening and closing versus if it's doing a grinding, which
gives you kind of an upsidedown U shape, or if it's just kind of doing food gathering, in
which case the jaw doesn't move; it's actually a tongue that's moving. So what you see is
more like a little dot. So the jaw doesn't actually move.
So in this case you can start to see under different conditions the pig is actually doing
different things.
So we do side-by-side comparison. So in this case we're looking at different runs and
looking at the teeth distance, so this is laying the model side by side for different
conditions.
We do overlaps. So we have two different types of ways to do overlaps. One is between
two different datasets, so the yellow set versus the green set, or we can look at different
cycles within the same data.
So, again, the cycle is defined as just an open and close, is considered as a cycle. And
this is actually really neat because if you just kind of plot out the cycles so that they all
originate at zero, what you see in this case is very clearly at least two different clusters of
motion. So you have the bottom one where it's kind of longer, the middle one, and then
you have a little outlier that sticks out. So this is actually comparing the same motion
sequence but looking at cycles as a time.
And this is basically saying, well, we're going to let the user play as much as they want,
and we give them as many comparison tools as we can think of, whereas George thought
of, and see if they can actually do comparison of the data using the tools.
And the last example I'm going to talk about, actually presented this paper -- Danyel and
George heard this, I guess, from this year's EuroVis -- where we're looking at principle
component analysis.
So a little quick refresher of PCA. It basically is a regression analysis of sorts where
you're finding the most dominant eigenvectors as principle components.
And in this particular case at the bottom what you show is a bunch of data points in XYZ,
and if you perform principle component analysis, you can find PC1, PC2, and PC3. But
in this case what we can do is just kind of forget the third dimension and re-project it onto
just the first and second dimension and you don't really lose that much.
The insight about PCA is that while I can teach an undergrad how to calculate PCAs -- so
it's kind of tedious, but you can do it -- it's actually really hard to understand this
semantically. So once I compute a principle component, I can tell you there's a linear
combination of the original dimensions, but what is the semantic connection between the
two.
So if I see a data point, how does it -- where is it going to show up on the plot, on the
projection. It's really hard to kind of map in your head. And this is not just for novices;
for most people that relationship is really hard to understand.
So what we seek to do is to say, well, so we have two different spaces. We have the
original data space and we have the eigenspace. And we're going to try to see if we can
use interaction to map between those two spaces so that you can understand what their
relationships are.
So this is the interface that we have. And I'm just going to show this as a quick demo.
So we have loaded the data already. It's an 8-dimensional data. And you can zoom in,
you can do all kinds of stuff.
But the key point I want to talk about here is of the eight dimensions we have a slider for
each one. So these are the original data dimension whereas this projection is on PC1 and
PC2. So the X axis is PC1, the Y axis is PC2. You can choose the axis as you see fit, but
let's just stick with these two.
So what we want to see is how does each original data dimension affect the projection.
So in this case what I do is I can change the contribution of one of the original data
dimensions. And you can see as I'm changing this how that affects the overall projection.
So in this case, when I'm doing this, you can see a rotational affect, but you can also see
that changing a little bit of this dimension actually affects the projection greatly, whereas,
you know, I just keep playing with this, and what you will find is there is some
dimensions that really don't do very much, so in this case it really doesn't do a whole lot,
whereas in some cases the dimension is really important to the projection itself.
So we propose that the interaction actually gives you some insight into the relationship
because you can see the rotational affect around some invisible axis, whereas if I actually
just show you as a static image, in this case I'm enabling trails, you really don't see the
same rotational affect. You still see kind of a rotational affect, but it's almost as if it's
rotating around some Z axis coming out of the screen, when in fact that might not always
be the case. It's not really that -- rotating around some Z axis, but it's something a little
bit harder to describe.
So we implemented this and what we wanted to see is if people can get a more intuitive
understanding about PCA and the projection of spaces using this kind of interactive
system.
So we compared the tool to a SAS/INSIGHT, which is kind of a commercially available
mathematical package, really, and the results that we find is that our system is a little bit
more accurate, not a whole lot more, but what's interesting is people don't actually give
up on the analysis. And I'll get to that a little bit later, and we also see that using our
system is not faster.
But people do like it better. So when we asked them to assign grades in terms of how
easy is this to use and to understand, they give us more As than the other guys.
And I'll talk about the results a little bit more later. But this is an interesting result
because we had envisioned that the evaluation will blow the SAS/INSIGHT out of the
water, and that was not really the case. So I'll get to that a little bit later.
But one of the key point that I need to talk about is I've been proposing that interaction is
the same as thinking, so you can use interactions to think. And this is a pretty strong
claim. So the kind of question that we need to ask is, well, what is in the user's
interaction. And if you really believe that interaction is the same as thinking, then can we
learn something from the interactions. Or, better yet, is it possible that you can take the
thinking out of the interactions and do something with it.
So to answer this question, again, take half a step back to think about what is in the user's
interactions. And if you kind of look at the model of a user and a computer, I'm going to
assume the computer here is a visualization, what you have is a very simple model where
you have the visualization on one side, the human on the other side, and you have the
person looking at the visualization, which is kind of in the form of an output or images
through the monitor. And when the user interacts with the visualization, you have this
input through mouse and keyboard. So minus the sound and all that.
This is actually kind of an obvious relationship but it's a pretty simple relationship
between you and the computer. That's it. You have the image, and you have the mouse
and the keyboard.
But if you kind of categorize how we usually use computers, I kind of roughly categorize
into three different things. So there's the kind of type of text editing, where you do a lot
of typing, you're not really reading a lot from it, you're just verifying it really. On the
other side, you have the output heavy where you're watching a movie where you click
play and you just sit back and watch. So there's very little interaction but there's a whole
lot of output.
Whereas if you -- the way that we designed visualization and visual analytics, that
relationship is actually closer to 50/50. So you're interacting, you're seeing the result,
and, you know, create a feedback and you interact some more and you see the result
more.
And in this particular section when I talk about what's in the user's interaction, I'm
actually specially focusing on the bottom one, so I'm not talking about the other two.
So we wanted to perform -- wanted to do a study where we want to prove that there really
is thinking within the user's interactions. So this is the goal that we set out to prove. And
the experiment's a little bit complex. You kind of have to bear with me a little bit.
So what we have is we got a bunch of financial analysts, and these are analysts from
Bank of America, from Wachovia, so all the big banks -- or previously known as
Wachovia, now is something else. When I ran the study it was still called Wachovia, let's
just say that.
So we got ten of these analysts and we asked them to use the wireless tool that I talked
about earlier to perform financial fraud analysis. So we gave them a specific task and,
you know, they tried to find fraudulent, suspicious activities using the tool.
So while they're doing this we're logging everything that they're doing. And I put the
word semantic in parentheses because we're not logging XY mouse positions or key
strokes, but we're monitoring which visual element they've highlighted or which thing
that they double clicked on. So it's actually semantically more meaningful, and only
meaningful to the application; it's not meaningful at all to other type of applications.
So on the other side we have a bunch of grad students that we called coders and we gave
them another tool that we developed for looking at interaction logs. And we fed the
semantic interactions into this interaction log. And then we asked the grad students just
to take guesses about what the analyst strategies are.
So we collect that and we go back to the analysts and said, well, what were you actually
doing. Right? So we showed them the video of what they did and we just asked them to
tell us what their strategies and what their findings and methods are.
And then basically we just compare them to see how close they are. So the key point
here to mention again is that the grad students have no knowledge of financial fraud
analysis. And they have -- they didn't actually see the analysts do anything. So they're a
completely separate group that have no interaction with each other. The only interaction
is me and the co-author where we do the comparison at the end.
So what we show is that the hypothesis is that if there is a -- if you can extract a lot of
thinking, then the two will be fairly similar. And we're actually really surprised by this
result. We were expecting that you can probably extract like 25, 30 percent. And as it
turns out, the numbers blow our expectations out of the water.
So in this experiment we find that our coders can extract up to about 60 percent of the
high-level strategies, 60 percent of the mid-level methods, and 79 percent, 80 percent or
so of what the analysts actually found.
And in particular if we kind of look at this a little bit more is that we wanted to look at
what type of things that they're able to extract and what type of things that we're not able
to extract.
So what we did was we plotted the participants, or these are the financial analysts, from
one to ten, and we looked at how well their -- how accurate their finding strategy and
methods could be extracted. And the two that immediately stuck out are these two little
low bars. So, again, everything was kind of roughly about the same but ->>: Sorry. What -- this is how accurate they are in ->> Remco Chang: No. Sorry. How accurate we -- how accurate the grad students ->>: [inaudible]
>> Remco Chang: Right. Sorry. How accurate the grad students are at guessing what
they were doing. Got it. Okay. Sorry.
And so the two things that really stuck out are these two little green bars at the bottom.
And the question is why are these so much lower than the others. So the coders were
only able to recover about 15 percent of these two analysts' methods, in particular their
methods.
So we went back to watch the video to decide, well, what happened here. And what we
saw was that the analyst strategy or method in discovering fraud was looking for spikes
within the data. So in this particular case they were looking for spikes in amounts.
Spikes in amounts over time.
So if you think about the only thing that the grad students had were interaction logs, they
don't have what the analysts actually saw, so they have no way of guessing that the -- the
reason that the analysts clicked on these transactions was because of the spikes, because
they couldn't see what the analysts saw; they only had the transaction logs.
So the understanding here is that just capturing the user's interaction in this case is
insufficient. So it's not enough to just look at the interaction logs. You actually have to
kind of get a sense of what they're actually looking at, being a visualization application
and all.
Yes, sir.
>>: I'm just -- what about some of the other ones? How were they so successful?
>> Remco Chang: So that's a really good question. I think why were they so successful.
So actually let me -- there's -- I'll get to that in a little bit. Hopefully I'll answer that
without answering it directly.
But, okay, thinking about what we learned from this experiment, the first thing we can
claim is that I think we successfully have proven that a great deal of thinking is actually
encoded within the interaction logs and it's extractable and capturable. Now, I did say
anything about using a computer to do it, but at least the information is there.
So kind of projecting a little bit further, we're saying that if you can capture this type of
semantic interaction and we can collect all this stuff and we can extract all these things,
then the hypothesis is that we can actually call this kind of a knowledge base. So this is
where I start waving my hand, because I haven't done any of this work.
But the thinking is that if the interaction logs do contain thinking and you can collect all
this thinking, then why don't we put it together for future use. So we're talking about
training, guidance, verification, and all that stuff.
But as we show, not all the visualizations are interactive. So you have visualization
where is basically watching a movie. And certainly not all the thinking is reflected in the
interactions. So what we start to realize is we need some kind of understanding about
what is in fact capturable, what is extractable, and what are the things that you can do in
terms of getting the thinking out of it.
And we start working on this and we submit a paper and it got horribly rejected. We got
terrible scores. So we kind of have to go back to the thinking board to think about what
this model might be.
So this used to say "in submission to InfoVis" this year, but it got horribly rejected, so
now it says I'm thinking about sending it somewhere else.
So kind of just to wrap it up a little bit -- I'm going way faster than I normally do, but
hopefully that's okay. So some of the conclusions that we can draw from this, is first
assumption is that I hope I have shown that interactions are important for visualization
and visual analysis.
And the first thing I talked about was in considering the interactions, we must be aware
of what it takes to make something interactive. And in the case of large amount of
information, we have to do some kind of simplification level of detail, approximation, but
the key insight from my own experience is that while you do all these things, you just
have to be really careful about what you're actually taking out. And a lot of times a
quantitative measurement might not be enough. So you have to consider the semantic
and the qualitative aspect of things.
And the second thing is I hope I've shown that interactions could be useful, and we have
shown that it -- it's useful sometimes a lot of times in different domains and different
applications. And in some cases the interaction is the difference in terms of the analysis
process.
And the last point capturing and storing analyst interactions we believe have great
potential. It could potentially be aggregated into a knowledge base, which it's really -- it's
really been pretty hard to do for a lot of the discipline. So some of the things that people
have done are ontologies, formal ontologies and so on and so forth, and a lot of those
things are either like giant trees of lots of different possibilities or whatnot. And this
might be a different way of thinking about what is a knowledge base.
So while I've talked all these things about what interaction is great for, I have to talk
about the things that it's not good for.
So, first of all, it's horrible for presentation. Right? So I'm sitting up here trying to tell
you interaction. I can't -- without actually showing you a video and actually playing with
it, it's really hard to convince you of such.
But more importantly what we realize is that when you have something, a visualization
system that's just openly interactive, you have this problem of your mileage may vary.
So reproducibility is a huge problem. Even within the same user, every time they walk
up to something like this, they do something different. So two different users, totally
completely different.
And evaluation becomes a bear because sometimes you discover some things, sometimes
you don't. So when somebody says, hey, tell me how accurate your system could be in
discovering fraud and using WireVis, I have no number because sometimes they do great,
sometimes they do horribly.
I myself went through a similar process where looking at the bank data I saw something
really bizarre. It was some company that's called Red Cross something or other, but it's
not Red Cross, and it's dealing in diamonds out of Russia. And I'm like that can't be
right. And I was so excited. I was going to show it to somebody. And I closed the app
accidentally. I had to restart it, and I couldn't find it again. I was digging everywhere. I
have no idea what I saw the first time. And the path was just lost. So this makes the
evaluation incredibly challenging.
And the last point is when we develop these kind of interactive applications, we often
sacrifice accuracy. So, again, the simplification process. In the iPCA, what we did was
instead of using the traditional singular valid decomposition we have to use an
[inaudible] approximation method that's called online SVD. We didn't come up with it.
It was some mathematical journal that somebody did this.
And it's not as accurate, but it makes the computation time much, much faster.
In the WireVis case, we realized that we can't really do clustering of large datasets. So
instead we used this really dumbed-down version that we call binning. It's just basically
histograming in some ways. But the difference is now you allow people to do clustering
on the fly, but then you have to do it in a really dumb way.
So it's this balance about speed versus accuracy.
And interestingly, as I talked about earlier, when we compare interactive apps versus
noninteractive apps, we don't see really a drastic time improvement. We do not see a
great increase in terms of task performance accuracy. But we have observed empirical
evidence that, first of all, the users are more engaged.
So in this particular case what we show is these two little green bars that stands for
nonapplicable are basically participants who used SAS/INSIGHT for a while and they
just say I quit, I can't figure it out, I have no idea what's going on, so they just gave up.
Whereas using more interactive apps, they're more engaged. So they're more focused and
they kind of just lost track of time, they just keep going until they actually find some
answer.
And, by the way, the answer might not be right, but they at least will go all the way to the
end. And, you know, we show that in terms of user preference, interactive apps are much
more preferred than the other.
But the last point is the most important one, but it's also the most subtle one, is that we
find that the users have a faster learning curve. So what that means is the first time they
do it they might be horribly slow, but as they start using it more, they actually learn the
problem domain in a much faster way than if they do a static application.
So the example, this is a separate study where we looked at interactive online maps, like
in our case we're looking at Google Maps and Google Earth, and compare it to the
traditional GIS application like the ESRI, ArcMap, ArcGIS, what we realize is that where
the user is spending most of their time is in exploration. So they zoom in and out, they
pan all the time. When you give them interaction, they use it.
So if you think about what they're doing during this playing around is that they're actually
building contextual information. So now they actually have some better understanding, a
mental model about the neighbor, about the community, about what's around the point of
interest, whereas if you have the traditional one where they just create a bounding box
and zoom right in, they don't gain the contextual information.
So what this means is if you ask them to do a similar task the second time, the people
who had the Google Maps would have this stuff where they had the exploration don't
have to go through that process again; they can jump right to the answer much faster than
the original one.
But what all this says to me is that, you know, right now we can't just measure kind of the
benefit of interaction using the traditional time or accuracy. There's got to be something
else. There's got to be new ways of testing it, new measurements or new methodologies
for doing this.
So kind of wrapping it up about future work, I'm really interested in urban visualization.
So one of the things that we're trying to do is to show a new type of map where it's not
just about the street layout but showing the characteristic of a city.
So this is the kind of stuff that when you go visit a friend in a new city he draws for you
on the little napkin and hands to you and says, okay, this is roughly where everything is
so you won't get lost.
This is the kind of map that we don't have. This is the kind of map that's often the most
useful but often we do not have these things.
So one of the future work for me is to actually look at how we can create what we call
semantic maps.
And we want to apply interaction in visualization in other domains, just to condition with
this line of thinking. And we have a couple of proposals accepted to look at studying
science policies with NSF to look at civil strife, so we're looking at causality between the
government oppression of people and the people reacting by having riots and see how the
negotiation between the two pan out.
And then we wanted to evaluate the benefits of interaction more, quote/unquote,
scientifically.
And lastly, about interaction capturing, what's so-called provenance, is we wanted to look
at semantic methods for actually studying the interaction logs. So so far we've just asked
people to do it. But we want to see if we can actually do this through machine learning,
data mining, and that kind of good stuff.
And we're looking for a collaboration with the Pacific Northwest Lab to look at a more
generalizable structure for recording the provenance.
And that sums it up for me. Thank you very much for your time. Really appreciate it.
[applause]
>> Danyel Fisher: We have some time to take questions.
>> Remco Chang: 15 minutes. Yes, sir.
>>: So maybe you could elaborate a little bit on that question before which was sort of,
okay, obviously there were cases where they just really didn't get it. But can we get at a
little bit why they were so accurate at all? What were the strategies that they could look
at?
>> Remco Chang: So I've given this talk about three, four times, and every time I've
given this talk, one of our UNC grad students was with me and they're able to kind of talk
about it from the coders' perspective.
And this is actually part of the problem with this study where I deliberately did not
participate in the coder process; I actually participated in the analyst process and
therefore I did not feel right to participate in the other.
>>: But you [inaudible] WireVis but you never showed us the interaction log, so I don't
even know what ->> Remco Chang: I understand. Okay. So this is a little bit hard to see. So I just have
to talk about this very high levelly.
What we have is this visualization with the X axis, again, time. So each row here has a
different meaning and a representation. So the top, the A part, is actually for annotations.
So in this case we don't have annotations, we hid the annotations. But the subsequent
three things represent each of the views in WireVis. So this application [inaudible]
because it's really tailored for WireVis app.
So if the user interacts with one of the views or with one view, it will show up here, with
the other view it will show up here and here.
And because WireVis is designed so that the relationships or the views are really
specifically focused on one type of relationship, you can start guessing. So if they're
interacting with a heat map view, then they're looking for a relationship between time or
keywords and accounts. If you look at the temporal view, then they're probably
interested in time and so on.
And here the bottom one, these two, show you the depth of the investigation, so how far
down the hierarchy tree are they looking at. And the last one is just looking at the time
range to see which time slice that they're actually zoomed in on.
So what I observe, and this is just my observation of the coders doing this, is they just
look for [inaudible] patterns. So, for example, when I show you that you can just kind of
mouse over keywords, and if they do that just kind of repeatedly, you just see a whole lot
of that on this graph, I believe.
So when you see that, then you realize that that's what the user is trying to do, is either
looking for the relationship temporally or in terms of how they correlate.
Now, what's important is where do they go next. So if they go to the temporal view, then
it tells you something. If they go to the different view, it tells you something. If they
start clicking on the accounts, it tells you something else.
So I believe these are what they are looking for. And one of the things I didn't get to talk
a little bit about here is that what's intriguing to me is that when the students did the first
participant, they were horribly slow. So if the analysts did it for 20 minutes, they would
take like 30 minutes to analyze the dataset. So it renders that -- this method completely
useful as far as time goes. They might as well just watch the whole video.
But as it went on, so by the tenth participant that they're investigating, they're super fast.
They're going through it in matter of minutes and seconds. So there's something that they
learned while they're looking at this and they're actually just kind of learning repeated
strategies from different people. And every once in a while you get some guy who gets
stuck because it's a new behavior that's never been seen before, then they spend more
time.
But overall what's intriguing to me is that what the participant -- or what the coders found
is not a fluke, because it's learnable. As they went on, the accuracy did not decrease but
the time went up, so they're learning something. Whatever it is, they're learning
something. So it's not just ad hoc kind of flukey that they found something.
Yes, sir.
>>: I was wondering for this direction log, right, I mean, it seems what people are really
picking up on is, you know, static activity where it's people do repeated activity in the
same part of the visualization, and then there's a transition to another one.
>> Remco Chang: Yes.
>>: So would it make sense to think of these interactions kind of in a network fashion
where you have nodes where people stay in one area and do some things and then they
transition to another area and look in a different [inaudible].
>> Remco Chang: Right. That's a really good point. Now, the assumption, however,
you're making, this is what we wrote in the paper that got horribly rejected, and I really
don't understand why. They just don't see my brilliance.
I think the implication about being able to do the capturing and talking about all the
semantic is that your visualization has to be encoding those things to start with.
So the question is if I give you three views that basically all do the same thing, then your
interaction log is not going to tell you anything because you can't follow along with that
type of thinking.
And you'd be surprised at how many visualizations have redundant capabilities between
views. So the fact that our visualization was designed so that they are fairly disjointed
makes it a lot easier.
So this is the lesson learned afterwards, that, you know, thinking about how to design
better interaction capturing actually implies designing better visualizations.
>>: Well, I don't know whether it's implying better visualizations, because you might
want the redundancy in the visualization because --
>> Remco Chang: Sorry. You're right.
>>: [inaudible] -- task better.
>> Remco Chang: Absolutely.
>>: So if your goal is to be able to capture after the fact ->> Remco Chang: Absolutely. So this is where I got horribly dinged in the review
process, because I basically said -- in really thinking about this I realized that there are
times where designing a visualization for the purpose of use versus designing
visualization for capturing, sometimes they actually clash. They don't actually fit into
each other, or least I couldn't -- I wasn't smart enough to figure out how they fit into each
other.
Because you don't want that redundancy. You actually want a single pipeline flow. And
you don't want to encode three different information in the same icon. You want each
icon to be disjoint, to be precise, so the interaction would be meaningful.
Now, if you think about all visualization design in traditional, kind of the tough D design,
they're basically saying cram as many things as possible into as little space as you can.
So that could make the interaction capturing really, really challenging. So if you click on
an icon that has seven different layers of meaning, why [inaudible] click on this. I don't
know. It could be one of the seven things.
And I'm being hammered on this, and I knew I was going to be hammered on this. But
go ahead.
>> Danyel Fisher: Actually, there's a question from George first.
>> Remco Chang: Oh. Yes, sir.
>> George: It seems to me that when you're trying to evaluate interactivity, you'd have to
take into account the nature of the tools that you're using for interactivity.
So I was thinking about, for example, the task of finding the needle in the haystack, the
literal task [inaudible]. The [inaudible] interaction would be to [inaudible] through that
haystack until you find the needle. That's obviously not the way we do it.
A slightly better way would be to use a detector and use that to [inaudible]. That's better.
An even better way would be to use a thing [inaudible].
So all three of those are interaction techniques, because the tool you're using to interact
with the data changes the [inaudible]. And somehow [inaudible] evaluation of what
you've presented, you don't seem to be taking into account the nature of the tools that you
use.
>> Remco Chang: I agree. And I think part of the challenge in the future research is to
see if it is actually possible to differentiate or to isolate the factors saying this is really
coming from interaction and this is not.
So in one of the possible future studies that we're planning to do was to take, for example,
the PCA project and fix out the interactions, so just lock it out, you can't do it anymore.
So we'd have the exact same app but with more of a query typing interface versus more
of a slidy bar direct manipulation type of interface. And do task and comparisons.
So we do want to isolate the factor and what we're trying to say, again, similar to the
spirit of this interaction study or this interaction log study is to say I don't really know
what's in it, I don't know what the strategies are, but I want to know if there is some
benefit of such.
So if I can just show some applications in which you take out certain type of interaction
and all of a sudden the user can do X, Y, and Z, then we can at least say that there is
some benefit to it.
Now, I absolutely agree that this is totally application dependent. But in kind of looking
around in terms of visualization and interaction research and evaluation, I just really
haven't been able to come up -- come across that many things that actually did the
isolation clearly to the point where we think we can use.
So it's a wide-open area, and I really don't have all the answers. I think it's like a huge
problem. This is really just tiny stab No. 1.
>>: So I guess we can also think about the visualization that you use for the visualization
of the -- or of the interaction. So, for example, you happen to choose a timeline view
that's oriented towards showing you which window users were clicking in.
>> Remco Chang: Yes.
>>: Which is useful and interesting. But it would be a very different view if you showed
me, say, which object data users were interacting with.
>> Remco Chang: Yes.
>>: Or ->> Remco Chang: Yeah. I totally agree. But you have to take a step back at looking at
what our original hypothesis and goal was. The goal was to say is there any thinking, is it
extractable in any way. And the fact that I can get 60, 80 percent out of it means that
there is at least something. I don't know what's there. I don't know what I'm missing. I
don't know how I can get to a hundred percent. But at least I show that there's something
in it.
Now, I hope that part of the research is to say, well, let's beat 60 percent and let's beat 60
percent not on WireVis but on everything. And I don't know what that is.
Yes, sir.
>>: I just had a -- some clarifying question on this slide.
>> Remco Chang: Yes.
>>: So what is it exactly that the grad students were writing down and the analysts were
writing down and how did you compare them to see if they were the same?
>> Remco Chang: That's a great question. So we had to break down an analyst process
into three areas. So what we realized is that we have this kind of a hierarchy of process
in terms of the analysis. The first is kind of a strategy level, so the highest level. It's
before they come into analysis. They have some preconceived notion about what they're
going to look for.
So in the case of -- so this a synthetic data. So we're now talking about real Bank of
America data. But they would come in, they'll look at the keywords, and they'll say you
know what? I think anything with Mexico and pharmaceutical is odd. Right? So that
would be, for example, a strategy.
Now, the second tier would be the methods, which says, okay, let's say that is -- anything
related to that is questionable; what do you do, where do you start. So you can either
look at accounts that have those two things together or you can look at the temporal
pattern in which those two things are close together, or various other things that you can
actually do to implement your strategy. So that's what we call methods.
And the last thing is finding. So finding is very precise. It's about which account or
which transaction actually contains suspicious activity that the person deemed to be
questionable.
So we categorize those three types of behaviors. I mean, this is a rough categorization. I
think if you look at analytical literature, like there's a much better hierarchy of sorts. But
we kind of roughly categorize into those three areas.
Does that answer your question?
>>: Not quite. I mean, that sort of tells me -- it gives me a rough idea what the concepts
were at the three levels, but, I mean, were the grad students -- I mean, were there like a
fixed set of codes and the grad students write down saying at time A I saw this code?
>> Remco Chang: Gotcha. No, no, no. So there's no code. It's all kind of just
handwritten. It's interviewed, so there's a lot of transcribing going on here where
especially like -- so these two processes are the same. So analysts telling us what they
did, you know, we're transcribing what they told us from words into literal things, and
there's definitely quite a bit of us as the comparison people to be as unbiased as possible.
So, like I said, the study -- there's quite a bit of hand waving. But one of the things that I
can say very safely is that we took a very safe approach on this. So I generally was -genuinely was interested in what I was going to find. So anything that do not match
perfectly we toss out.
So, for example, if the analyst says my strategy was looking for Mexico, pharmaceutical,
and Brazil, and our student says, oh, Mexico and pharmaceutical. So if they miss Brazil,
we don't give partial credit; we consider it as wrong.
So we take a very -- I don't know what the right word is -- but basically anything that's
not perfectly right we consider it as wrong.
>>: Okay. So the interaction logs actually do contain the data elements themselves, not
just they were looking in this window, but they were looking at this window and focusing
on both Mexico [inaudible].
>> Remco Chang: Right.
>>: The data is encoded in the interaction logs.
>> Remco Chang: Yes. So which visual element they're highlighting or which visual
element they interacted with, that's what we're storing. We don't care about the XY
position of the mouse.
Yes.
>>: There's been a lot of research on the benefits of animation in comprehension. And -yeah [inaudible]. And I'm just wondering if you're never going to get to a hundred
percent, because possibly the design of your user interaction isn't quite right?
>> Remco Chang: Yeah, yeah.
>>: And doesn't quite match the task perfectly and leads to misunderstanding or
confusion?
>> Remco Chang: Absolutely. And I think even if you take 20 steps back and -- you can
say that a lot of analysis actually happens in someone's head. And unless you have some
FMRI thing or something -- so one of our research goal was to say, well, let's be kind of
practical about this.
So we have to imagine an environment, so the environment could be -- let's say you had
the bank. So you can't attach all these external thinks, you can't look at giant, you know,
30 cameras around the guy's desk, you can't have an FMRI stuck on his head -- or, sorry,
EEG stuck on his head or FMRI sitting [inaudible]. So all of that is out.
>>: [inaudible]
>> Remco Chang: I don't think the bank would let us. Yeah. But you're absolutely
right. So like I was saying to Danyel, part of this research was actually just to answer our
own question that says we've seen a lot of people try to do interaction capturing that fail.
And I hate to talk about this because it's almost like, you know, bombarding other
people's work, but the PNNL Glass Box was -- according Jim Thomas [phonetic] was a
complete failure, was a multimillion dollar failure. I know that [inaudible] tried to do
something that didn't work out. Google tried to do something and it didn't work out.
So there's a lot of people who tried to do this that did not work. And the first question we
had was given all the failures and very little success as far as we were able to find, is this
even reasonable to pursue. Can you actually get a thinking analysis high-level strategy
just by looking at interactions.
So that was definitely our step one. And I think once we've proven to ourselves that
using the most kind of manual way, there is something that we can think about, well,
what did we do wrong, can we improve that number and so on.
>>: You actually touched on this before, so [inaudible] what do you think makes this
particular analysis so successful in getting [inaudible].
>> Remco Chang: You should have been my reviewer, because that's what I wrote about
and nobody seemed to care.
But so I think the -- what I start today realize, like I said, was we wanted to get good
results in the capturing, at least our own experience is it starts with designing a
visualization.
So, again, we didn't design it thinking about capturing, we designed it kind of two years
beforehand. But whatever it is that I did made every visual element very straightforward.
It doesn't have multiple meanings in the same visual element. So I don't encode hue and
transparency and whatever all into one GLIF, right, because once you do that, you click
on it, and you have no idea why the user clicked on it.
And I did not do a horrible amount of redundancy. There is some redundancy between
the views, but there's, you know -- the semantic meaning behind each view is fairly clear.
Right. So when you have that kind of isolation in terms of both the window space or the
design space, then the visual elements themselves, then any sort of interaction with these
things become very obvious about what they mean.
So what I realized about the success of this is not really what we did as an experiment but
really was kind of an indication that the original design of WireVis was really suited for
capturing. It's not initially implicated as good visualization, but it's really good for
capturing.
So the question is is this generalizable, and I don't have an answer. And does it mean that
other people should design their visualization to be capturable? I don't really know. But
I think you can't separate the two. You can't say that we're going to have good capturing
techniques without thinking about what the visualization does.
All right. Is there any other questions? Thank you so much.
>> Danyel Fisher: I'd like to thank you our speaker once more.
[applause]