>> Jonathan Grudin: Okay. Thank you for coming to the talk today. We'd like
to welcome Conrad Albrecht-Buehler, if I pronounced that correctly.
>> Conrad Albrecht-Buehler: Yep.
>> Jonathan Grudin: Who I met -- I visited Northwestern, where he just
recently got his PhD working with Don Norman and others and I visited
Northwestern about a month ago and afterward had a conversation with Conrad,
just prior to his actually defending his dissertation, where I realized that the
overlap of his interests and interests in groups here was very high. And so when
I got back I invited him to come and he begged off on showing up until after he
defended his dissertation on Halloween. But he did that successful and now he's
here to talk about Heed, which is a framework for Situation Aware Monitoring and
I think also some other things. So thank you very much.
>> Conrad Albrecht-Buehler: Thank you. Thank you, Jonathan. Thanks.
Thank you all for coming, I appreciate it.
What I want to do is I want to tell you four user stories about monitoring and
some of the difficulties that these people -- that each of these people have. I'll
discuss the situations that are important to each of them and how they each
perform their various monitoring tasks. Each story is an example of a monitoring
task that competes with many other responsibilities for these people. And is one
for which alarms and dashboards don't always work.
So the first story I'm going to tell you about is about monitoring servers,
something that I think to some degree we all probably have a little bit of familiarity
with.
The second story is about monitoring business performance by reading financial
reports. The third story I'll tell you about is about monitoring the user's
experience in a discussion forum.
And the last story that I'll tell you about is about monitoring for infectious disease
outbreak at an animal shelter.
And I'm going to try to make the case that alarms and dashboards are essentially
incomplete. I'm going to base that on a distinction that I'll make that between
monitoring activities that are analytical and those that are evaluative. In other
words, those that try to reach an objective conclusion, compared to those that try
to reach a subjective conclusion.
So let me start out by telling you a little bit about a CTO that I work with. He's a
CTO of a small entrepreneurship and the company's business is offering services
online. So it's a very technical company and he's in charge of everything
technical there. He's the lead developer and so he's created the website and
most of the application that runs on it.
He also has executive duties which involve raising funds and doing interviews,
but he's also the systems administrator for the entire company. That means that
he has to build and maintain all the computers, including all of the servers.
So one of the things that he monitors is the performance of the web, the
database and the mail servers. So to do that he has these powerful server
dashboards that give him a huge amount of information. Now he ends up rarely
using them at all. In part it's because it's a lot of work for him to find what is
relevant to him in these. I mean, they can be enormous.
And they end up requiring quite a significant amount of some mental effort to
spot and assemble all the data that is relevant to a situation that he wants to be
aware of.
One of the other things that he has at his disposal are alarms and alerts to let
him know that an incident has occurred. But he ends up filtering all of these alert
messages that come in and put them into their own mailbox, which he basically
never looks at. He really only looks at them if there's been an incident and he
wants to sort of try to trace back to try to figure out where it's origin was.
So if he doesn't really use his alarms and he doesn't really use the dashboards,
how does he even know that something's ever wrong with his server? Well, he's
got two ways.
For one, sometimes he'll get someone from his customer support team will come
by and tell him that they're getting angry phone calls about people who can't
access the site or can't do something on the site. In some ways he let's his
customers do his monitoring for him. But throughout the day he's also working
with the servers. He ends up, you know, applying patches, updating application
code and so on. And whenever he happens to be logged in into the service he'll
run up-time just to check the server's load.
In essence what he's doing is he's sampling the server load. If he happens to
catch it at a moment where the load is high and if he has a little time, he might go
and investigate the situation and try to understand what is happening. So in
other words, he's essentially asking is the server load high enough that he needs
to go investigate it. But high enough...what is high enough? That's a pretty
subjective description. What might be high enough for me is probably not high
enough for him.
Let's take a look at a quick graph of server load. So is the load high? Was it
high? Is it now? Will it be soon? A lot of questions about how the server load is
doing. Why doesn't he just set an alarm? Well, for one we already know he
doesn't really use them. There's another reason. He wouldn't necessarily have
time to drop whatever he's doing in order to go and investigate why the server
load is high. And when he does have time, an alarm doesn't necessarily tell him
anything. It would just be silent if it's not high.
So the problem is that this is what an alarm does.
It divides the data range. It divides it into the data that he should ignore and the
data he should attend to.
Now remember that he'd investigate if the load was high enough. Alarm just tells
him that it's high. And what's more, it tells him that once it's happened, nothing at
all before it happens. The graph would tell him that a little bit more. He could
see how it's changing. He could see where it is, where it's been, but it's at a cost.
He has to interpret the graph. He has to spend time looking at it and trying to
understand it and then he's already dismissed using his dashboard for pretty
much the same reason. And it also takes up a lot of screen real estate and as
we saw with the dashboards that can balloon really quickly.
So instead what if we change how the data range is divided up from this to this?
Instead of showing the whole graph, we could just show the region prior to the
alarm threshold. Then the position in the region could identify the importance of
attending to that situation. If the data is below the "ignore" threshold it might as
well be at the "ignore" threshold. If it's above the "attend" threshold, might as
well be at the "attend" threshold and somewhere in between it tells them how
important it is that he might go look in on the situation.
So I prefer sort of this range in between as Heed and it enables us to indicate
this sort of degree of importance. And since our CTO wants to know when the
load is high enough at that time, we don't necessarily need to show him the
whole graph, we could just show him its current position in that range.
And so now we could offer him a simple dashboard that relies on the way he
already monitors his servers by looking at load. And one that doesn't give him
too much detail and still will help him decide if he should turn his attention to
those servers. And it does so continuously, so when he has time he can quickly
identify if at that time it's worth investigating. And by making this dashboard tiny,
easily and persistently visible, he can evaluate the situation quickly and develop
some awareness of how his servers are loaded, how frequently they get loaded
and so on.
So what would end up happening when giving him this little dashboard that we let
him use for a few weeks? So he ended up learning quite a bit about how his
servers typically perform and that ended up increasing his confidence in their
performance and their operation. And he explained to me how it ended up being
used to actually catch an incident that could have been a problem, but it wasn't
his to catch. He had -- since this thing was running in his office and his other
developers go in and out of his office and meet with him from time to time,
eventually enough had asked questions what this thing was and he explained it
to them.
I didn't get to teach them anything about it, but he did. And he told me about one
incident where a -- one of his developer his walked by his office and peered in.
And he wasn't there, but the developer looked at the display and saw that the
database server had a high Heed and it suddenly reminded him that oh, he had
left a database backup running and it was the middle of the workday so he ran
back and quickly stopped the database backup so that the customers had more
access to the site.
So even -- to some degree that ended up showing that this was a relatively
simple to interpret display and something that even his developer could use and
then he explained -- the CTO explained to me how much he had wished he had
this display several weeks before. They had gone through a major upgrade and
they had replaced the entire web application. And despite having done months
of testing once it had been launched there were these instances where the site
would just grind to a halt and they would look in and they would see huge
amounts -- tons of processes running on the database server.
The web server was getting bogged down, but they didn't understand why. And it
wasn't consistent, it would just happen sometimes. And this was such a problem
that he ended up having to dedicate himself and three of his developers full time
for three days watching constantly what was happening on these servers in order
to catch the moment when this problem began so that they could try to
understand and interpret it. And he had said how much he wished he had a
display like this to give out to everybody in the company so that everyone could
be watching for a time when one of the developers needed to go look in and see
what was causing this problem when the load was increasing on all these things.
Ethan?
>> Question: Are you going to talk about longitudinal data? Like one of the
neat things about the chart that you had earlier is if you're willing to roll back and
look at the logs, you can see that historic -- or you are watching dashboard, you
can see that historical trending.
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: It feels like here I can say, oh, the database load is really, really
high ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: -- so I don't know that it's only very recently become high or it's
been high for the last three hours because I went and took a long lunch after
asking the database to back up?
>> Conrad Albrecht-Buehler: Right. That's actually -- that's a great question. I
will actually get to that in just a moment. There's -- essentially what you're asking
is a person monitoring may actually care more about how something is changing
than where it is. And so we'll get to that in just a moment. But it's also -- the way
in which you actually asked the question is actually I think very telling. It's -you're doing an analysis of what's going on and a lot of people who monitor
things, especially very technical things, will tend to do that. We want to
understand what is going on. The only thing is that we don't have a way to
identify when we should do that. And that's what I'm trying to cover primarily.
So let me tell you a little bit about the next user. She -- so deciding to investigate
the situation with the servers with the CTO is a tactical decision and it's one he
may have to do several times a day. But what about problems and situations
that you need to keep track of that happen far less frequently?
So the CEO, she's the executive of a medium-sized business that also offers
online services and she's launched several programs to grow the business. And
she needs to keep an eye on how those programs are performing. However, she
doesn't need to look in on them every day, every hour, she just -- once a week or
so is enough for her to get a sense of how they're performing. But like our CTO.
She has way too many other responsibilities and constantly sort of dealing with
much more urgent and immediate problems.
As a result she forgets to checkup on these programs and in fact told me that
one time that she hadn't looked in on them for a month and a half, only to find
that they were underperforming tremendously. Furthermore, these programs
carry a lot of anxiety for her. She's consider worried about their performance and
actually as a result that only discourages her from checking up on it, a little bit of
avoidance.
So the way in which she monitors these programs can she reads these reports
that are generated for her and if the report shows that a program is not
performing well enough or based on her expectations, she might go investigate
and try to understand why something isn't -- why users aren't signing up or why
they're not signing up faster.
In many ways this is a lot like our CTO's problem, except that the values don't
change as frequently. They change a little bit every day. They change a little
more every week.
But as a result since they're close enough, maybe we can apply similar solution
and just show her how close each of her clients are to being underperformers.
Now we could just show her these four indicators and let her know how each of
the clients are doing, but what happens when this grows to 20 clients or 40
clients? We really want her to read the report and spend her time analyzing the
report, so we need a way to sort of compact this display so she can just decide
whether or not she needs to go read that report.
So to do that we can just assign some values to the Heed range and just say, set
0 for the minimum Heed and set 1 to the maximum Heed and make it a scale.
And now we can elect the maximum Heed value to represent the whole report.
That means that instead of saying client M needs your attention, it says that the
report needs her attention. And it also says that unless the whole program is
performing well she should probably go and investigate the situation.
So this is essentially an administrator operation applied to these four Heed
values. Now she has a single representation that summarizes the report and
again if we make it persistently visible she can continuously evaluate the
program's performance.
Also of note, because we're just assigning these values to the extent of the range
we're -- we can combine data of any kind. It doesn't matter whether it's numbers
of users or CPU load. We can simply combine them because we're just saying
how important is it that you pay attention.
Furthermore, because we've sort of compacted this display in a sort of
predictable way, we enable her to delve into this and into the situation. Meaning
that she can get an evaluation report and then she can delve in and get a little
more detail about the -- to get sort of a more detailed evaluation report and then
she can go and look at the very detailed actual report and raw data, which is
essentially the sort of analytical process. So there's an evaluation that triggers
the analytical process and then there's this act of analyzing what's going on.
So working with the CEO actually had very surprising results. The biggest
surprise was -- well, it was very surprising that when I would go to interview her
after leaving the application running with her she would -- she would tell me that
she had -- she would apologize and say she hadn't used the display she'd been
so busy and had all these things going on. But when I questioned her about the
performance of the program, when I questioned her even just about the position
of the display, she knew exactly where it was. Despite the fact she didn't think
she was using it, she ended up using it. And the fact that it was this very almost
passive interface was a really surprising result, something that I certainly didn't
expect.
But the biggest surprise was the fact that it ended up reducing her anxiety over
the performance of this program so much. It wasn't that the program was
suddenly performing so much better, but she was aware of how it was
performing. And she asked me to create many more displays for her, not
because these were necessarily things that she needed to monitor, but these
were aspects of her business that were creating a huge amount of stress for her.
So she was looking at this as a way of decreasing her anxiety, which I thought
was a very surprising result.
So the third story I want to tell you about is about a community website owner.
There are some situations that people monitor not because they're already
familiar with them, but because they want to learn about them. This is one such
case.
Now he runs a small website -- well, medium-sized website now with map, which
is primarily a large community-supported website where people can interact and
inform each other and it's largely just a huge forum.
He's also the primary developer for this site, he created all the software for it. He
created the whole site himself, but the site runs on co-located servers, so it's not
really his responsibility to go monitor the status and know how they're doing.
He's more interested in had knowing what his users' experiences and trying to
make sure they have a good one. The thing he's looking out for is if they're
having a bad user experience.
And he talked to me about two types of negative user experiences he was trying
to look out for. Something that restricted access to the site for his users and the
discussion on the site becoming what he called unbalanced. So let me start by
telling you a little bit about the access restriction problem that he saw.
So since he's the sole developer, he's frequently changing the site. He's
changing the site code and he told me about this instance where he had made a
small typo essentially in the code and a subset of his users couldn't log in. And
at a different time where a subset of his users couldn't post to the forum.
And so he was trying to monitor for this happening again. The way he found out
the first time this happened was he happened to be talking to a friend of his
who's also a forum member, who then told him, oh, I haven't been able to access
this site for a few days, is something going on? And he had just had no idea.
Even though all the data about the numbers of people logging in, the number of
people posting, the number of people applying, were all -- was data that he
actually in theory had access to because they were part of a sort of Ruby on
Rails framework that he used to create the site. The only way he really had of
seeing this data and querying it was through SQL queries and he had enough
other things to do that wasn't something he'd end up looking into.
So what he wanted to know was if he had something else that ended up reducing
the ability for people to log in or reducing the ability for people to post or reply.
The thing is that there might be a day when there's a lot of people posting, a lot -it's been a day where there was a lot of -- some national event happened which
everybody had an opinion so there was a hot topic and everyone was replying or
it was a day after a holiday and everyone was talking about something that was
important about the holiday, so there were many new threads started.
He doesn't -- it's not that he cares that neither people are not posting or replying,
it's that -- I'm sorry, it's not he cares that people are neither posting nor replying,
not just one or the other, which is essentially an "and" operation of those two. It's
not a logical "and" which would sort of try to pick the minimum of the two values,
but it's a compensatory measure instead. Because if people -- not posting
doesn't mean that they can't post so maybe he should look in on the problem.
But if they can't post and they can't reply he definitely should look in on the
problem.
Now this issue of monitoring for unbalanced posts, unbalanced discussion is a bit
unusual. He talked about making sure the user sort of felt comfortable,
especially when new users came to the site. And an unbalanced discussion
forum he felt was one where people would be reluctant to want to contribute,
especially as new users. And he described that as happening when there were a
lot of posts, when there was just a ton of activity where it just felt so -- it might
feel insulated to somebody who just came to the site. And so it might be too
much -- a time when there's just too much activity he may want to go in and
moderate the discussion a little bit.
And the other thing he wanted to look out for, what he called Dud posts. These
are when somebody posts something, but to which nobody replies. In other
words, it's essentially kind of like an announcement of some sort. And that's a
problem that he felt would maybe make somebody feel like it was a dead forum
in which nothing was really going on. So he was looking out for this, but how did
he really know that this was the case because he'd, you know, gone in and
moderated before?
Well, he used his gut to make that decision. He participates on the forum
himself. He looks in on the forum F. It feels like there is too much of it, then he'll
step in and do something. But it relies on him being a party to the site and being
constantly involved. But more importantly he has -- he didn't have any idea what
these numbers were. He didn't spend any time with those numbers. So the
previous two solutions involved knowing what some of these values were, having
familiarity with the data. So how was he going to even find that out?
So what we did is we started out with the same sort of solution, but we used a
very conservative estimate and just said maybe it's a very huge range. And if he
had got high, he would go and investigate. He would say, okay, I'll go see if this
discussion is sort of getting out of control. If you find, oh, no, my gut telling me
this is fine, he would then go and adjust the value. He would modify the value to
reflect what he was -- what his interpretation was of the information. And really
what this interaction amounts to is just moving those thresholds. If -- I'm using
the graph from the CPU load to illustrate this, but since the data value is fixed,
the pulling the slider down is like moving that -- moving the "ignore" boundary up.
So as -- I'm going to walk up here and show you this. So the data value is fixed,
so pulling this down is like dragging this "ignore" boundary up. Likewise, if he
were to move it up and say, you know what, there is something going on here
you're not telling me. I'm going to make sure that next time this happens this is
an important situation. He drags it up. It's actually just bringing the attend
boundary down. So after a while he would do this and it would end up reflecting
his experience with the website more and more.
But to answer your question, the thing that he really wants to look out for is that
it's this problem is building. That it's not just that it is getting close, but that it is
building so he can nip it in the bud. So that is exactly where the graph would
come in handy, like Ethan had pointed out.
But we still want to -- oops. We still want to use the sort of -- try to make it an
evaluation, so we'll add the same sort of boundaries and like I said before, once it
dips below -- once the data is passed, one of the boundaries might as well be at
the boundary.
The other thing is that a graph, as we said sort of before, that there's too much
detail in. It just needs to make a quick evaluation, so a lot of the detail's
extraneous and so we can down sample the data.
>> Question: (Inaudible) --
>> Conrad Albrecht-Buehler: Oh, it fell down. Sorry, Henry, thanks. So we
can down sample they'll data and we end up with a much more simplified graph
of what's going on. But we also want to be able to predict how soon he's going to
need some attention and so I tried to offer him a sort of forecast of what Heed
would be in the near future. I tried to look at some really clever learning systems
that would do a good job with long-term data, but wasn't successful so I just used
a common filter to at least give them a sort of short-term forecast of the data and
change the display to reflect both of that, to show the current value, the history,
as well as the forecast, and represent occupancy likelihood in this sort of grid
with a gray scale.
But this display is still -- takes quite a bit of screen real estate and its biggest
problem is that the current value is a bit -- not included, but it isn't really obvious.
So we want to know he still needs to know what the current situation is. So I
created this little animation sort of shows the relationship between this graph
view and the sort of slider view we were looking at before. And shows, highlights
the current state.
It shows how the data and changing and how it's likely to change in the future.
It's intended to sort of convey motion, sort of similar to that cartoon, that method
of drawing comic books where a ball is flying through the air and your little motion
lines behind it, this is intended to sort of evoke that, as well. And users
consistently interpreted the direction in which something was changing pretty
readily. And so now this also, the display sort of adds another step to that
delving subject. The subject can -- the user can see what the situation is, what
the evaluation situation is, and then expand the display, sort of get a sense of
how it's changing before then digging in deeper.
This is some actual data from the access interference test that we did with the
user in which we were looking for posts, but what is sort of interesting here is that
I've all shown the 10 boundary at the top and the ignore boundary at the bottom,
but in this case he care when is people aren't posting. That was the issue of user
interference and so it's actually important when that number gets low, not when it
gets high. But the -- obviously it's trivial to invert the actual display and make
sure that Heed is always oriented with high Heed at the top for sort of a
consistency to maybe better see how this historical data relates to the actual
data.
If we flip the graph it makes it a little bit easier to see how it's down sampled to
this. But this inversion was a little bit confusing for this user and for a few others.
So we added a little trick to sort of -- a little interface element to remind them
what was high Heed and what was low Heed. I hate the red-green mapping, but
it was what the user really remembered and could interpret. So we just used
these little reminders, but after a while, he, too, disabled them.
So our form owner ended up learning a lot -- learning quite a bit about his users'
behavior with this. He ended up finding -- he would make these adjustments and
then actually go in and look at the configuration file and see where these values
were and actually find out how much people are posting and it gave him a much
better sense of what was actually going on at the site. So much so that he ended
up building his own version of the display that I had nothing to do with. He just
simply rewrote the configuration file. He instrumented his website even more
and tried to use it to learn about all sorts of other aspects of his -- of the website
and how it was used, which was a pretty sort of exciting result.
So the last story I want to tell you about is about the Chief veterinarian at a very
large animal shelter. So there's some situations people wish they could monitor
for, but they're just too complicated and this is one of them.
The chief veterinarian also had a great many responsibilities. He's not just a
veterinarian that has to care for the animals, but he's also an administrator and
he has a lot of sort of budgeting duties and personnel duties. He also has to -ends up testifying in court cases in which he's had to talk about animal abuse, so
he has an enormous amount of work to do. And this is a very large animal
shelter.
The problem that he's particularly concerned about is infectious disease outbreak
at these -- at the shelter. The sort of infectious disease outbreak is a problem,
not just because he cares about the well being and welfare of the animals, but it's
very expensive when there's a disease outbreak. So he wants to try to identify
the fact that an outbreak is about to happen and try to nip that in the bud, as well.
So the reason outbreak happens is because that the shelter the animals are kept
in close quarters, which unfortunately can spread the disease, but is necessary
because they want to try to rescue as many animals as possible. They also, as a
result, have very limited space for isolation. Any room, any kennel that is
reserved for isolation is another kennel in which they can't rescue an animal.
But, the animals get sick from time to time. They'll catch -- they'll come in with an
infectious disease to the rescue, but it doesn't mean an outbreak is going to
happen. It just means that an animals sick. An outbreak is when it starts
spreading and there's rapid infection between many animals, which is something
that's definitely happened to them in the past.
If the disease seems to be spreading, there is some mitigating actions that he
can do. He can call for thorough sanitizing of a room which the animals are kept
and which they call nuking a room. But before he calls for that, which is both
very expensive, it's very time consuming for the volunteers of which they never
have enough anyway. He wants to be certain that all the animals that are in that
room are not carrying the disease, either because they've been infected and are
now immune or haven't gotten the infection yet.
So he wants to be certain of that, but it's a really difficult problem for him to even
determine that, which has been part because the -- many of thee infectious
diseases have symptom-free gestation periods. So some of these gestation
periods can be a week and a half long and the animal will be contagious to other
animals, but nobody -- but nobody knows they've got the illness yet.
Even if somebody notices that there are some symptoms, the animals are
necessarily immediately brought for care and there's so many volunteers that
information doesn't really get around very easily. Furthermore, even if an
animals cared for or diagnosed by one of the vets, there's so many vets that all
those people don't necessarily communicate that kind of information with one
another. So the way in which he would know is he'd have to actually walk by the
isolation room all the time, recognize there's an animal there and recognize it is a
new animal being there.
So the other way in which he could know is he could go through the records that
they have for veterinary care, but there's a lot of them and he does not have time
to go through the new records every single day. And lastly the problem gets
made each more complicated because the animals get moved around. They are
in one kennel one day and they get moved to a different one on another day.
The volunteers have to sort of make these decisions based on some animals are
not getting along with one another or one seems to be particularly uncomfortable
in one kennel. So they get moved around.
So just the act of actually investigating this problem is really difficult and very
time consuming. So he needs to know when it's valuable to even go investigate
this problem where he's going to end up dedicated quite a bit of time just to figure
out if there is an outbreak about to happen or beginning to happen.
So essentially the importance of him going investigating this is based on the
number of incidents that have occurred in each room. So the display we made
for him tells him if the disease is spreading through the shelter by showing him
how many rooms are showing infection, are showing likelihood of an infection in
there.
If an animals diagnosed with the illness every room it's been in for about the last
10 days needs to increase it's likelihood for being infected, but adjusted for how
many days since it's been there. To determine all this we need to know which
animals have been diagnosed and then where they all have been. So luckily that
kind of information is stored in those veterinarian records and the animals sort of
private records for the -- at the facility and those are stored in an online sort of
medical system. But it's a proprietary system and so the data isn't really readily
available, but we can parse it by hand and luckily one of the volunteers was
willing to do that for us.
So if each incident sort of increasing the likelihood of the room being infected
after several days that likelihood diminishes because now either all the animals
have gotten sick or they're likely already immune. So if the likelihood gets high
enough in any room then it's worth his time to investigate that situation. And the
more rooms that get high and the more likely it is that he may have an outbreak
on his hands and he may have to make time to investigate and make some is
very important decisions.
So what's interesting actually about this is that he didn't read the display as time
for me to go investigate. He knew, he recognized that, but he also said doesn't
that also tell me the danger that any of these rooms are at infecting new animals
going in?
If I could just put this information outside of every door then all the volunteers
would know and when they would actually maybe take preventative measures,
they'd be more care envelope that room. They might decide there's enough
danger in that room that they won't put a new animal in there, they'll put them in a
different room. And I thought that was interesting that he ended up mapping this
sort of Heed to something else, in this case to danger. And he saw it as a way of
actually not just increasing his own awareness, but increasing the awareness of
many other people involved in the site at the animal shelter.
So from all these stories and from all this work, this all leads me to describe what
I think is needed in order to monitor a SWAG effectively. First the observer or
the person who is going to do the monitoring must identify and describe the
situation of interest to them in terms of some set of indicative conditions.
Next they need a way to quickly perceive that situation as an evaluation in some
way. And then they need a way to investigate the situation. Delve into it and
analyze more. And finally they need a way to refine and modify that -- their
understanding of the situation there or define and modify the situation and base,
to reflect their changing understanding or the changing situation.
I've sort of tried to make the case that with each of these stories that alarms and
dashboards and reports aren't sufficient, that they're analytic tools and in the
case of all four situations the users valid benefited from or do benefit from some
kind of additional tool to support making evaluation.
The evaluation is necessary because all these people have many
responsibilities. They must constantly make decisions about where to direct their
attention. And the closer a swag is developing, the more important it becomes to
pay attention to it. And so tools that aid in making those evaluations quickly will
help make monitoring easier.
Thank you. Thanks all for listening. Thanks for tuning in. Thanks.
>> Question: I was going to cover this earlier, but how is this distinct from a
dashboard?
>> Conrad Albrecht-Buehler: So for one thing that it simply becomes this
thing, you can create, you can compact the display on and on to make a single
indicator. Let me show you, I think you may have missed the slide in the
beginning about -- no, no, no, please.
So in this case we can, this was a report in which an executive was looking at the
performance of several clients and you could just show this is a giant dashboard
or we can actually create a process by which we consistently refine -- summarize
them all into a single display that becomes an individual indicator. And the sort
of framework -- actually I didn't go into a lot of the detail of the things I showed
the "or" and an "and" operation, but with many other logical operators that allow
you to just describe much more sort of detailed descriptions. Shouldn't say many
more operators, two more operators, one is not.
It is very important for a user to be able to describe what's typical and then say, I
care when it's not typical, which is just a matter of essentially inverting it, taking
one minus the value. And then they need sort of censor operators, not s-e-n,
c-e-n-s-o-r operators to be able to say, I care when this happens, except when
this other thing is happening. So they need -- so I have something called "and
not" which isn't exactly an "and", but it works by essentially suppressing the
highest Heed value and it takes the minimum of these values and allows the user
to define the situation in terms of mitigating circumstances. That's how it's
different than a dashboard, I don't know if that helps, but it also requires less
mental effort to parse. So ->> Question: The server example, the massive dashboard versus the indicator,
also shows the ->> Conrad Albrecht-Buehler: The dashboard shows raw data and this is
intended to show an evaluation of the data.
>> Question: What you could do on a thing called a dashboard ->> Conrad Albrecht-Buehler: Right.
>> Question: They didn't -- you largely created that for all these people as they
have this new dashboard thingy that they can look at that is a lot more evaluative
than purely analytical thing they used to have to provide.
>> Conrad Albrecht-Buehler: Absolutely. And, you know, the most effective
dashboards are the ones that sort of highlight what's important in them that draw
your attention to the stuff that's happening in that -- that's being shown in the
dashboard that might be relevant to you. But in order to do that the dashboard
has to have some sense of relevance ahead of time and so the people who
develop the dashboard can do a great job of capturing what is going to be
relevant to the users of the dashboard by studying those users. But this allows
the users themselves to define it.
>> Question: My question unless it is outside this particular range this,
particular area, never care about (inaudible) unless it's too high.
>> Conrad Albrecht-Buehler: Yeah.
>> Question: So you bring all this, note, there is trouble.
>> Question: Also, following up on that point and it's kind of interesting, can you
comment on how important the cropping actually is? Humans are actually
relatively good, in fact, amazingly good at looking for ranges and interpreting, you
know, given that you've already compressed it to single dimension ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: -- what if you just list it from 0 to 100 and you didn't crop the -- I
think you were calling it a Heed region.
>> Conrad Albrecht-Buehler: Yeah. You mean compacting it sort of just down
to that one line and then just leaving that whole ->> Question: So you do two compact ones as far as I can tell, right?
>> Conrad Albrecht-Buehler: Right.
>> Question: One is you compress time and you get rid of time in many of
these cases. And then the second thing you do is you crop the "ignore" and the
top and the bottom.
>> Conrad Albrecht-Buehler: Yeah. Exactly.
>> Question: And I'm particularly interested in how much you gain by cropping
the top and bottom off as opposed to just letting everything go really high or
really low because that does provide you with more information.
>> Conrad Albrecht-Buehler: It does, absolutely. But when the information
that it is preparing you -- I'll walk over here. The information it's providing you is
essentially analytical in nature. You're already, if you are being told that it's
gotten super high you're apt to start trying to interpret it, trying to understand it.
This is actually just trying to let you know that you need to go find out if it's super
high or not. That the distinction is about how -- when this sort of activity is taking
place. Is the activity first taking place in your -- deciding this is how you're going
to spend your time or you are going to go to that important meeting. Then the
activity is about analyzing the problem, understanding it, understanding how high
it is, is that a serious problem, all that. That involves quite a bit of sort of
cognitive effort. You want to be able to sort of distinguish those two and that's
where these -- what I'm sort of advocating for is that we need something that
helps us make the decision to go do that to start interpreting it.
>> Question: So maybe the follow-up question to that then is for each of those
domains was there really a threshold at which I would go dip into it or -- because
most of the scenarios I would envision would say it's always a trade-off in time
whether I go attend to something and there's a continuum of, this thing is like
shooting through the roof, I'm going to drop everything I'm doing right now and go
attend to it or if it's just above the threshold I can probably wait five or 10
minutes.
>> Conrad Albrecht-Buehler: What I would say is that if you've said that it's
just above the threshold then that threshold isn't properly set, then it might still
need some adjustment. You're basically saying it's not important enough for me
to drop everything to do it, that means to me that it's actually below sort of
maximum Heed and if approximate you're saying you care when it's really
shooting through the roof, then I would actually say that it's not so much about
making the decision based on the actual graph of the data, but actually making
the decision and setting the thresholds in the first derivative of the data. So
basically creating a new censor from other censor values.
So the thing is that people actually are really good at -- in my experience actually
describing what they want to pay attention to. I didn't show this, but I found that if
I asked a person both of these questions, inevitably they could answer one of
them. They could just tell me what are you looking at, what are you looking for.
By answering what are you looking at, it's basically allows me to sort of help them
define the situation that's important to them in a bottom-up fashion. Starting with
the censor they are looking at and sort of adding mitigating circumstances to it,
mitigating conditions to it. And if they could tell me what they were looking for,
they were describing the situation like unbalanced forum. What does that mean?
And then they would start describing the conditions that led to that, the conditions
that were meaningful to recognize it was an unbalanced forum and so then we
could build that situation description in a top-down fashion.
>> Question: And kind of this reminds me so much of (inaudible) small multiples
and this example the hospital chart that he did where there are many, many
censors that he presented on normalized -- normalized each one then ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: -- and actually did display a time series instead of selecting a time
series, we would have (inaudible) ->> Conrad Albrecht-Buehler: Well, I have to admit that my recall for Tufty's
work in small multiples, I remember the picture with all the t-shirts on it and not
the one you're talking about, so I'm afraid I'd have to actually go back and look it
up a little bit.
The importance of the time series is that in that case it may have been that it
allowed -- I'm only speculating here because I don't remember, but it may have
allowed the nurse or the doctor to make the interpretation based on how much
things were changing. That again this was about including first derivative into the
sort of decision-making process. And by -- in this case you can actually further
compact it by basically saying that first derivative is just another censor in this
case and we can add it to that.
But the thing is it's not necessary to compact it. It's actually fine to show it just in
a full dashboard with lots of these and then look -- letting a user look over them
or even sorting them based on how high the Heed is. So here's a miscellaneous
one, many situations, and subject can look over them and just identify, you know
what, this one is not only high, but it is increasing really rapidly, I should go look
in on that one. It's not necessary to always compact them and maybe just as
valuable to sort of set up a sort of mini-dashboard or even a larger dashboard.
I don't know if that answers your question sufficiently.
Patrick?
>> Question: I was wondering how you deal with changing -- in the cases
where you have maximum threshold it's obvious, but in this example you showed
the "or" searches --
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: -- you were sort of mapping 0 to 1 to the maximum value, but the
maximum could be changing from one day to the next.
>> Conrad Albrecht-Buehler: Uh, yeah, absolutely. So essentially what you're
asking is that those maximums are changing because the situation is changing,
the personal understanding of the situation is changing ->> Question: If I understood your picture right, I thought the maximum was like
how many users have locked in or are registered in the system or something like
that.
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: There could be a day where there's 100 new users and your
maximum is higher than the previous day. Right?
>> Conrad Albrecht-Buehler: Absolutely. So the situation is actually changed.
The world has been somewhat changed. So that is actually another problem that
we have with alarm systems that in fact there is a great study by some people in
the U.K. in Bambory(phonetic), these colleagues. And what they found is that
alarm systems fail in a lot of these large industrial plants because they're set for
these values that are not actually appropriate to that factory or to that situation as
the factory that was moving through more and more product.
And so they actually advocate it's very important that you have a system by
which people can adjust their alarms. So what you're talking about is maybe on
one day it was perfectly fine that, you know, 50 users logging in is plenty and
that's not a problem, but the next day there was a newspaper announcement,
tons of people joined the site and now he's going to care when les than 100
people log in. And so the thing is the adjusting of the slide allows them to
actually adjust those so to continually try to match what his experience is with the
system. It says essentially that that's a never-ending process.
>> Question: (Inaudible) ->> Conrad Albrecht-Buehler: Well, to some degree, but that would be -- to
some degree the whole problem with alarms is they're telling you, go do this as
opposed to saying, go look into this. So maybe it would be more about
identifying when they need to go look in on maybe adjusting the sliders as
opposed to saying, drop everything, go fix them.
But some problems don't change just hour to hour. They change day-by-day.
They take quite a bit more time to sort of change and the understanding of these
it takes longer to change, but absolutely. There is this sort of problem that it's a
never-ending adjustment process. But it just happens in their mind already, it's
just is there a system that ends up enabling them to reflect that and sort of work
as a cognitive artifact that helps remind them.
>> Question: So you did talk explicitly about this, but I didn't see any
visualizations around it.
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: But it seems that at least some of these scenarios you have sort
of this bimodal thing where I care about if my log-ins drop below this number ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: And I care if they go above this other number.
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: So there's almost an "ignore" band in the middle and pair of Heed
regions and then a pair of alert regions.
>> Conrad Albrecht-Buehler: Exactly.
>> Question: Did you explicitly describe any of that?
>> Conrad Albrecht-Buehler: I didn't explicitly describe that, but actually I can
definitely tell you about it. So that's actually just an "or" combination of those
two, the high and the low saying, you know, I care when things are really high or
when things are really low. So just may go an "or" combination, the to and
(inaudible) like the higher ones. So when the data's gotten way up high then it
will -- so basically it means that in between the two ignore boundaries it will be
well for both and as it moves through one of the two attend boundaries it will
become higher.
>> Question: So when you say it will be low for both, you mean it will be just
basically right down on the bottom?
>> Conrad Albrecht-Buehler: Yep. Exactly.
>> Question: Yeah, that's a good example to get across the difference between
the evaluation and analysis. You don't get anything from -- you get very little just
from that one data point ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: -- from the evaluation or for the analysis.
>> Conrad Albrecht-Buehler: Sort of the big takeaway I wanted to really push
is that people have to make -- constantly have to make these tactical decisions
about how they're going to use their time, where they're going to put their
attention. And we need tools that help guide that, but don't force it. So like we
said with alarms that it's not about telling you go do this. It's about saying maybe
go look at this.
Frank.
>> Question: I've already mentioned sort of (inaudible), but one (inaudible) -which direction are moving away?
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: For you this is one obviously (inaudible) direction on this and
show us where is the (inaudible) very fast or very slow and which direction they
are moving, just kind of ->> Conrad Albrecht-Buehler: No, that's actually a fantastic point and actually I
purposely avoided adding any other color mappings because I wanted to sort of
leave that open for some future work. I think there's actually an opportunity
exactly like you said to sort of add additional information about how fast things
are moving based -- using a color mapping to maybe identify things as sort of
hotter and colder. But -- and which direction they're going.
But there's an intention here that they're always oriented the same way so that if
you sort of read left to right and up and down, you can see things that are sort of
moving this way, which give you a sense of how they're moving. But the
question of how fast they're moving is actually a very important one. And
something that I've avoided a little bit here.
So I didn't tell you how -- what the time distance is between here. There is a
dimension of time, nothing sort of said in between here. And some cases that is
something hour by hour, day-by-day, minute by minute, second by second, it's
not there. Now the actual display ended up actually having some labels here that
indicated how much time there was in between them. But the thing is that two
different Heed displays next to one another could have different time scales. So
one that seems to be growing really rapidly might actually be just growing over
the course of a week as opposed to one right next to it that is just changing
second by second or may not seem to be growing very quickly. And making that
distinction is very important, but it tends to map to what I found is that users end
up having a real problem with that. They ended up knowing that the problems
they were paying attention to had a time scale, they knew that these things
change either very quickly or very slowly and so they -- that didn't bother them.
But it's still actually a very important point and SML that will have to be
addressed more sort of in future work looking at this and trying to understand
how to compare things that really do happen at different time scales and
compare them together.
>> Question: So to follow-up of that. If you feel that treads are important, then
the main thing you're benefiting from in order to get a representation is space
compression.
>> Conrad Albrecht-Buehler: That's right.
>> Question: Could I get you to speculate on perhaps behavioral differences
you might have expected. So I don't know if you did, sorry, but (inaudible)
person because it looked like in a couple of these cases that they are very rich,
but they went from having no representation in the data to your representation of
the data.
>> Conrad Albrecht-Buehler: Yeah.
>> Question: Could you speculate if they got, you know, the traditional -- the
timeline representation of the data ->> Conrad Albrecht-Buehler: Yeah.
>> Question: -- would you expect behavior would have been different, too, in
reacting to the information?
>> Conrad Albrecht-Buehler: I -- I can't -- well, I'd like to try to speculate, but
I'm certainly not sure TOCHLT some degree it matters if the person is monitoring
something very technical and it also matters essentially sort of what their role
with regard to the information is.
So for example, the server, the CTO is also the systems administrator. He will
respond to the graphs very well. Much of his work involves looking at this. He's
involve involved. Looking at this, he's very involved in the very detailed
knitty-gritty. But the CEO who works at a much higher level, she wouldn't
necessarily benefit from the graphs and in fact from my experience of working
with her, very few of her reports have graphs. They have some numbers she
looks at and she'll sometimes copy them out into Excel and maybe do a couple of
comparisons, but not very much. So in her case I don't think that she -necessarily the graph will have -- a graph or more traditional sort of display would
be just as meaningful. But I don't know for example with the chief veterinarian,
whether he would benefit more from having sort of graphs showing how
frequently there were infections and how often that got spread and so on
because I don't know -- I assume he has quite a bit of technical training and must
use them similarly, but I don't have any way to know.
>> Question: It is a very nice compact representation.
>> Conrad Albrecht-Buehler: I think it's actually -- what I think is kind of neat
you can end up actually mapping this representation to other things. So for
example, just sort of off the top of my head, if you imagine a cell phone
dashboard that when you bring it up it shows you a bunch of different
information, but it shows you it all relatively consistently. What if the battery
indicator got bigger as the battery got lower? But I may be a person who just
drives between home and work and I've got a charger everywhere, I don't really
care until it gets down to 5% or 10% so I might want to sort of shrink it, but other
people who travel a lot more, who have to get on planes, they care more about
being able to have that battery. And if it is getting low or getting down to 50%,
then maybe it should be bigger and sort of though them that more, give them the
ability to sort of make those adjustments or changing.
Just like having, you know, you have an appointment coming up, maybe that
should be much bigger if it's sooner. Yeah?
>> Question: Situations where you think that this would not be appropriate?
>> Conrad Albrecht-Buehler: Absolutely. Absolutely. I think it's not
appropriate for data that changes incredibly rapidly and has to be monitored
extremely carefully. I mean, offhand I don't have any particular example to give
you. I'd like to -- yeah.
Well, actually the stock market is an interesting example.
That comes up often. People say, could you use this for the stock market and
I'm not sure because I don't -- I'm not an investor and I don't understand how
often investors have to look in on a portfolio. I know that you can set stop orders
and stuff like that, or stop loss orders and stuff. But it seems like it is an
extremely analytical process and so the relationship of investors to that data can
be very -- so analytical that maybe this is not meaningful.
>> Question: I think there are some opportunities here. I don't study that very
closely, but some of my friends do and I've seen them, I think there are Heed -there definitely is Heed areas based on past performance, but if it's involving a
certain range over the last five years then somebody enters into this, so I think
there is some potential there.
>> Question: I wanted to follow-up. Do you think the displayer should increase
the situation awareness or in fact decrease situation awareness?
>> Conrad Albrecht-Buehler: So what I saw when -- I saw is it seemed there
was an increase. And the way in which I knew that was first of all to some
degree define situational awareness. I'm sorry, the title sort of pumps me and we
have to do that, but I skipped it. Situation awareness is this field of study that's
been primarily promoted by Mike Enzoy(phonetic) and situation where she
describes sort of these three levels of awareness that the first level of situational
awareness is just sort of knowing the numbers, knowing the data. The second is
sort of being able to understand sort of what it means and the third is sort of
when you can predict and you can see what sort of coming down the pike, the
awareness that something's gonna happen soon.
So there was definitely -- there seemed to be in some cases there was increase
in the first level of situation awareness, like with the form owner that he didn't
have any idea what the numbers were, but he did know sort of what his -- he had
a level two sort of situational awareness about the -- what was happening on the
form because of his interaction, that was sort of his gut. He was actually using it
to increase his level one situational awareness. The user who, the CEO and the
CTO both actually increased their sort of level two situational awareness. They
understood better about how these things were doing. And then the vet actually
was using it in some ways to actually look forward. In some ways he had
increased his -- his level three situational awareness.
The biggest problem with situational awareness at this point is that it's still really
hard to study and to know for sure that awareness has been increased. And so
you've got to try to sort of look for indicators, but so that is why this is
somewhat -- isn't quite so numerically studied.
Did that answer your question?
>> Question: Yes. Sort of. I was just thinking like in certain critical situations,
you know, a person's ability to react to a changing situation is really key and
important.
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: And study like auto pilot situations, it's very dangerous because
pilots actually have to fly somewhere and it's about the status of the plane.
>> Conrad Albrecht-Buehler: Yeah.
>> Question: So when an alarm goes off, like, gosh, what is going on.
>> Conrad Albrecht-Buehler: Yeah.
>> Question: They don't really know.
>> Conrad Albrecht-Buehler: Yeah. Exactly.
>> Question: I imagine the same thing happens with this, once you hit a certain
threshold, unless you really understand the mappings underneath it, you don't
actually know what's happening in the situation.
>> Conrad Albrecht-Buehler: So one of the things that's going on here is that
it's actually, I'm not saying get rid of dashboards, get rid of alarms. Use this with
them so there still is a need to be able to say, you know what, there's -- pull up
the plane, which I always think is so interesting about alarms and airplane
cockpits. They don't just go off, you know, especially the collision avoidance stuff
actually gives you a command, which I think is really interesting, as opposed to
like a smoke alarm.
So now I lost my train of thought. Oh, I lost it. I'm sorry. I was answering -- oh,
yeah; right. Sorry. Something getting serious enough that you've got to respond.
So there's still a need for alarm. You still need an alarm, but hopefully this
actually allows you to maybe interact with the alarm system that this ends up
helping define those alarm thresholds and it will let you know when something is
going to happen or that something is getting close to happening so that you've
got some idea beforehand.
>> Question: (Inaudible) derivative seems like a crucial aspect of that.
>> Conrad Albrecht-Buehler: Uh-huh.
>> Question: Or a likely ->> Conrad Albrecht-Buehler: Absolutely.
>> Question: Can you expose the underlying algorithm that computes the
(inaudible)?
>> Conrad Albrecht-Buehler: That computes the ->> Question: Yeah.
>> Conrad Albrecht-Buehler: Yeah. It's -- it's going ->> Question: Like number of posts divided by replies or if it's for ->> Conrad Albrecht-Buehler: Oh, it's not even -- well, no, it's just setting
thresholds, it's just setting the line and then doing a transformation between.
This is just what function you use to transform in between isn't defined. It's up to
the user. So for example the gas gauge, it's not as important -- it certainly
doesn't decrease linearly that a change between half a tank and one tenth below
half a tank is not the same as the change between, you know, the last 10%. So
you might need a sort of exponential function instead. But this might look familiar
to other people, but to some of you, because this is essentially the same
definition as a fuzzy set.
So in some ways this is applying fuzzy set theory to interaction. And these -- this
transformation function in fuzzy set there is called a hedge. And on the same
sort of hedge stuff applies.
>> Question: (Inaudible) -- this one (inaudible) ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: (Inaudible) ->> Conrad Albrecht-Buehler: Uh-huh.
>> Question: (Inaudible) ->> Conrad Albrecht-Buehler: Yeah, absolutely. It's a wonderful way to sort of
classify sort of lots of different things. You can use this sort of as a matter of
classification in that way.
>> Question: (Inaudible) -- function.
>> Conrad Albrecht-Buehler: Uh-huh. (Laughter) Yeah. Well, it really could
be anything. One of the things -- one of the things this function can be is it can
actually be based on something very abstract, like the words that people use. So
that if in a -- you can just make a giant mapping of different words that sort of
require more attention than others, so if people are just talking about cameras
and gadgets or whatever, it may be low, but if people start talking about bombs
or something, maybe it's more important that you go pay attention to that so that
you can apply -- you can end up using just sort of any function. But it's -- it's an
evaluation.
>> Jonathan Grudin: Okay. If there's no more questions I'll thank Conrad
again and anybody who wants we're going off to lunch over next door if anybody
wants to join us it's fine to join us.
(applause)
>> Conrad Albrecht-Buehler: Thank you. Thank you. Mapping of different
words that sort of require more attention than others, so if people are just talking
about cameras and gadgets or whatever, it may be low, but if people start talking
about bombs or something, maybe it's more important that you go pay attention
to that so that you can apply -- you can end up using just sort of any function.
But it's -- it's an evaluation.
>> Jonathan Grudin: Okay. If there's no more questions I'll thank Conrad
again and anybody who wants we're going off to lunch over next door if anybody
wants to join us it's fine to join us.
(applause)
>> Conrad Albrecht-Buehler: Thank you. Thank you.