15841
GREGORIO CONVERTINO: Okay. Good morning, everybody. Thanks for
coming.
I am about to complete my Ph.D. this summer if everything goes according to the
plan and I've been at Penn State for the last four years.
This is more or less in a few words me. Geographically I was born in southeast
of Italy, Alberobello, that's where I spent my childhood, and half of my life was
basically in Rome and that's where basically I did my high school.
I graduated in psychology, bachelor and master, and then I went for an internship
in College of IST in HCI, on mobility, and then I decide to go back to the
university and study computer science afterwards in Rome.
After bachelor work I got the opportunity to come to the U.S. with a Fulbright
scholarship, and I spent my first year in Virginia Tech with Carroll and Rosson,
and then after the first year they decided to move to a different university so I had
to move with them. So I have been at Penn State since 2003.
In that (inaudible) I came to California for a couple of times for internship, one
time at PARC and one time at IBM Research.
>>: Which groups (inaudible)?
GREGORIO CONVERTINO: I was with Peter Pirolli at PARC, and with Moran,
Tom Moran at IBM.
So what I'm going to tell you today and there is probably more stuff than what I
can say, is two pairs of studies, the first pair is on awareness in CSCW systems,
and second is in grounding, information sharing specifically.
So let's start on awareness. So if we look at the literature, how is the concept of
awareness been defined. So going back basically at original studies of
face-to-face collaboration, sort of flight control type of studies of the late '80s, as
people work together for a long time in the same place you basically start
observing that they seamlessly try to align their activities together so that's what
originally was defined as awareness.
So in CSCW I see -- I sort of decomposed the problem as a design problem on
the one hand and a methodological problem on the other. So designwise, the
problem is that current systems are not supporting awareness very well,
specifically in the context of distributive and long-term work. So generally if you
look at the main shift once you move into the distributive work, you generally
have some distributive process lost just because people have to compensate for
the lack of a series of -- even so this is just a number had. So what we know
from the literature is that the more awareness you have, the less need is there for
explicit coordination so for less overhead somehow. So that's quite clear from,
you know, literature, from (inaudible) and others and so on, media space
research and so on.
So what we see currently as the problem is that in current system when you have
work distributed and when had it long term, a lot of breakdowns occur, a lot of
overhead is there, so that's where the problem is designwise.
Methodologically I see a couple of problems. So far there is little validated
methods, and when I say this I mean basically methods that you can repeat over
and over and you can be sure that you're still measuring the same thing. And in
definitions people tend to use definition of around awareness very sloppily and so
they tend -- I mean in the words of Smith and Robertson, the term awareness
has become a placeholder for very different phenomenon in reality.
So the effort here is try to operationalize a bit more the concept of awareness. At
least for what I will present here today I define awareness at a level of activity so
think of a group of people working on a grant proposal, a group of you, or
academics, for a long time, they have to coordinate work and so on over a long
time period, and what we define awareness in this context is understanding and
managing interdependencies between people, task, tools and situation, situation
as extend our resources and so on. So basically understanding and managing,
the terms are at the core of what awareness is.
How -- the way I approach the problem is try to use multiple methods. Now, in
this sense more generally I mean in a (inaudible) sense try to use multiple
research strategies, try to integrate results from the field with whatever I model in
the lab, try to grab the lab model over the field, and when I study things into the
lab I try to make the effort to have a multiple measure for each construct so that I
can increase a bit of reliability of the measurement.
With respect to the concept, the idea is to try to operationalize the concept then a
map into observables, into measures, okay, so that we see -- we see visually
basically what we are talking about.
Theory driven, I simply mean that if I outline the conceptual model and then I
translate that conceptual model into measure, an empirical model, then basically
I'm going to use the first conceptual model which is basically my summary of
bio-literature to interpret my result and possibly that feedback into that model and
refine it.
So given this approach, this is what the first two pairs of study that I'm describing.
So the first bullet is basically what the group was doing, when I joined it, they
were studying collaboration in school settings so this would be the teams of
students were basically working on a semester project and there were two
classrooms and groups across those two classrooms were actually supposed to
coordinate a science project together and so what the result of the field
generated generally was a classification of factors of why they would encounter
breakdowns. So what were typical factors that will bring them into breakdowns
and therefore lack of awareness.
So based on those classification, I will present the first lab study that was trying
to model some of those situations, some of those breakdown scenarios, into the
lab, and the second lab study is going to look more in-depth at the measurement
of awareness per se.
So as I say, what we pulled out of the field was a classification of reason why
people will start breaking -- lacking awareness was basically, you know, we'll
need recovery set to action to get back to what they meant or what was the state
of the work and so on, and we came with the classification of four factors were
changes due to the partner, changes due to the task, changes into the tool and
changes into the situation. Think of situation as external resources, lacking
external resources and so on.
So given those reasons or factors for breakdowns, we try to model them into the
lab and how we did this. Well, we created script scenarios based on what we
had observed into the field of why people would start having breakdowns, and
have one confederate into the lab that we would try to reproduce that scenario
using a script. So we will have six scenarios, one or two for each one of those
factors.
So the main questions here for this first time we created these lab models was
are we able to reproduce realistic work into the lab and, secondly, if those events
that were clearly not the same into the field, if we are still observing into the lab
that those events are still not visible. So those were the main objectives of this
first study.
So basically the method of this first study is the following. There were six
participants, a very small study, with six scenarios which are our manipulation.
We are trying to introduce those changes into the setting, and what we are
measuring is people behavioral awareness of this. So they either can be aware
of those changes, I'll give you an example of what those are, spontaneously.
They are not -- not seeing it, we have a script for basically creating a prompt so if
--sort of bringing the attention closer to the change and still see if they are not
seeing it or not so either they are aware after prompt or not. If they are still
unaware, we classified their response as unaware, very, very loose classification
at this point since it was the first study.
So you see this is basically the distributed setting and the participants just
balancing gender here.
So here is an example of what a scenario looks like. So we have observed this
into the field. We had this problem of some groups of students will work together
over long term, what will happen, one of the two teacher who basically is sort of
an external actor, will decide just to shift two dates, for example, in the project
manager, and on the other side they were completely unaware of this.
So at some point during the project they will just go into confusion just because
that change or that swap of dates into the project manager was not
communicated on the other side. So we simply reproduce this. So this is what
the confederate knows due to a change in the teacher plan of the class, is the
dates of the two activities in the project manager have been swapped. So this is
the history of change.
>>: Which kind of breakdown is that?
GREGORIO CONVERTINO: Situation.
>>: Oh.
GREGORIO CONVERTINO: Because the teacher per se is outside the activity,
right? So is sort of the external situation objectivity, right?
>>: Okay.
GREGORIO CONVERTINO: So it's very loosely saying external context with the
activity. Where the activity, I mean actors towards that goal of writing that report,
okay?
So what works basically do we use. Well, at that point, 2002, 2003, 2004, the
state of the art was more or less the group work space so that's where more or
less what the CSCW system at that time for collaborative editing would look like.
It would have a buddy list, a planning tool like a project manager now you see
where you assign tasks to people. There's a chat tool at the bottom, and this is
more or less -- and you can navigate through the (inaudible) panel, and what we
were doing in this situation, the task for them is an environmental science project,
so they have a bunch of data, they have to make sense of those data, and, you
know, define the problem and find solutions to the problem. That's what the
students were supposed to do. By the end they have to produce a report, okay?
So that's why cooperative editing.
So procedure, their first session, now, pay attention to the fact that in this first
study we are just reproducing collaborative sessions, okay? We are simulating
individual parts of the work, so the sight to do and they split the work, then we
assign to them individually between the session what were the parts they agreed
to do individually.
So in the first session they get some background information, they'll learn about
the procedure of the task, and the partners, they get some training on the work
space, and then each one of the four sessions they will do some organizational
work.
Yes.
>>: To be clear on your design, so you said they six tasks or scenarios for six
pairs of people?
GREGORIO CONVERTINO: That's correct.
>>: So each pair had their own independent scenario?
GREGORIO CONVERTINO: No, all of the six were actually given to all the
people.
>>: 0h, okay. Okay.
GREGORIO CONVERTINO: Yeah, it's (inaudible).
>>: Okay. That's (inaudible).
GREGORIO CONVERTINO: Well, in this study it's not yet repeated the measure
so each change will occur only once for each one of the members, okay? Okay?
>>: So six different tasks, six different pairs of people.
GREGORIO CONVERTINO: Yes, each person will receive six changes, yes,
and we check how much they are aware of each one of those six changes.
Okay? So 56 in total basically if you want cases of changes.
So what did we notice from these experience? So basically back to the
procedure, every time they do some cover operation, and at the end of the task
back to the -- in this case there is a single measurement for each one of the
changes, and at the end also we checked their perceived level of awareness by
the end of the project, and they're debriefed about the situation, they're told about
the confederate and so on. So in between the sessions they received the
individual work.
Now, what did we notice? What were the main finding of this first study? I
presented these in 2004. Basically major markers we were looking at how, you
now, how realistic is this, are they really doing the real work? I mean, they were
clearly engaged, the student will try to steer a decision towards their, you know,
their own strategy and so on. We will actually see pretty creative work, you
know, new solution that we did not expect, and in general what really was the
core of our interest here is are they aware of those changes or not. Well, in more
than half of the cases they were not aware even after one prompt. So it's still
true that in a work space those type of changes are very difficult to notice with
one prompt. Okay?
Yes.
>>: Were there certain classes of changes that were like the students go
unnoticed or -GREGORIO CONVERTINO: Good question. No. At this point, no, but we
cannot say that clearly because very few instances to actually claim significance.
So the next study we tried to do a bit more observation on that.
But even in the next study I can already tell you that there was no significant
changes in all the classes. I can tell you more about that.
But so it's really true in more than half of the cases there, they're not noticing
those events. Now, well, what about time? Well, it's true, obviously, as you
would expect, the more they are into the fact, the more they get experience, the
more they probably become able to notice those changes. So you sort of see a
linear -- more or less linear thread from 15 percent to 50 percent, depending on
where those changes were occurring within the sessions. So perhaps this is an
important factor, parameter that we immediate to take into consideration for the
next study.
And so three major -- just so we learn that we, you know, they have markers of
real work, but then we decided to actually do some refinement to the matter just
to make it -- to improve it a bit more. So we say that in the first study we were
just simulating the individual parts in this situation, also in the next study we
decided to also have people doing individual part of the work also within the lab.
So to make sure that we did control for how much individual work they would do
as well.
With respect to the changes to the scenario, we also felt that there was a bit of a
limitation to have only changes within the session. In other words, I know that
my partner is just putting the work in the wrong tool and I just don't know where it
is, but sort of, you know, that's an interesting event that causes a breakdowns,
but the more style changes are those that occur between sessions.
So where actually something is a mismatch with the state of the work space from
the last time, and that's very hard to remember what was -- what was the work
space from the last time and the Delta between the two. So we decided to
introduce multisession scenarios for the next study which were the harder ones.
And finally, you know, in former communication, we found it was pretty
interesting. So basically incidentally we found the more they will have informal
chatting, the better will be the quality of their work. So we thought the next time
to actually encourage some informal communication.
>>: You're saying that was an observation the first time or is that a hypothesis -GREGORIO CONVERTINO: No, it was just incidental that we found that pairs
would have more informal chatting would actually have the quality of the work
also better.
>>: In the first study?
GREGORIO CONVERTINO: In the first study. Incidentally, we did not expect
that, we just found it. So we decided in the next study to encourage informal
communication, you know, sort of chit-chat at the beginning or the end. In fact,
people will actually become very much attached, we had the problem sometimes
because they would ask for the IM of the confederate and we didn't really know
how to handle that sometimes because they would bond a lot towards the end.
So at the end it was a bit of an ethical issue to tell them look, you have been
tricked. But, you know, that was very natural in the sense that they will actually
find -- be attached to the person but, you now, sharing informal -- but at the same
time the quality of the work was very good because they get attached to the
project as our own project, right?
So second study what we are trying to do here is trying to stretch a bit the model
and make it a bit more systematic, okay? So two system rather than one, we are
trying to generalize a bit across some properties of the system. Still four
sessions in this case with individual work as well, and we are splitting the
scenarios into two types, I can tell you more about that, and what we are
measuring, in this case we are going to repeat the measures, so what's your
awareness now, what's your awareness now, first session, second session. So
that we see sort of the, you know, the trend of how much is changing, and some
additional control variables in metacognition and personality, just to see if there
are people's variables affecting their ability to be aware.
So, in other words, three main things we are manipulating here, the session -you know, basically the way we segment the activity over the four sessions
individual and collaborative about -- over a period of about three weeks, and they
have about one hour of work individually, one hour of work collaboratively, and
then they have a gap of at least two, three days again and so on, and so the
entire thing lasts about three weeks, okay? So that's the size of the project that
we are able to -- obviously, we cannot match exactly to the 25 weeks of the field
study, but we are trying to get as close as possible, and this is very expensive,
believe me (inaudible).
So the two systems have functional equivalence so they are both able to, you
know, support navigation, communication, they both have chat, in this case the
Bridge which is our research system, and Groove again, which is the commercial
system. They have a planning tool in both of the cases, collaborative editing, so
what really changes is the organizational work space rather than the tools
themselves. Okay?
So in some sense each one of those functions is supported somehow is the way
this is supporting those changes. We wanted to see if that is going to affect their
ability to become aware of those changes, and finally those were the scenarios,
back to the questions you were asking before, at this point we have assigned one
scenario for each one of the factors, back to a partner situation, and we have,
you know, the supposedly simpler one which will occur within the boundaries of
one session and those that will occur across session.
So with the multiple session think of a situation where the structure of the report
keeps changing. Every time we meet, you reshuffle the sections of the report
and I just don't know it for example. So those are the harder ones so we want to
really see if they are harder.
So my computer is a bit slow, but basically I'm going to show you how I
conceptualize this. So the idea is to start from a conceptual model where we can
specifically boost our research question inside the conceptual model. So the
work system is made of the setting, the tools, the people and the task, those are
sort of the static aspect that constitute the work system, right?
So then we have some dynamic aspects of it which is the temporality of it, you
know, the session we decide to have that much collaborative and individual in
that type of intervals and that segmentation, and those our scenario that will have
the incursion into the work setting with that specific schedule so we know that
type of change will always occur in that session in that point in time, okay?
Situationally at least.
So the question here is how do we measure awareness which is an aspect of the
interacting group. So that's the first research question that we had in mind as the
core, and the second is are there people's variables that will affect their ability to
be more or less aware at this point of the activity level, and as I said before, if we
are comparing the two systems, are there tools -- more precisely would actually
be system aspects that will affect their ability to be aware or not, and if they're
highly aware or not, what are the consequences on the product, are their product
better or not. Okay.
So let's attack the -- these are just a different way to look at the variables. What
we manipulated is on the left, what we measure processwise is in the center,
what the outcome -- notice that I have -- I try to have subjective and objective
measure for each one of them, okay? Just to see if they tell us the same story.
So those were the results. We had a pretty large questionnaire which I
constructed expanding the questionnaire from the first study and trying to have
different aspect of awareness in clusters of -- and we found that of the six cluster
coming out of the question of 59 items that were the clean ones, one factor was
basically loading on 62 percent of the variability of that. So we use that as the
proxy of our measurement of activity awareness, okay?
Now, if you plot that over the four sessions, what you notice is interesting is that
there is a warmup time, so people do need -- consider that we're measuring this
at the end of the session. So by the end of the second session, they still don't
increase in this perception of awareness. So they do need at least a couple of
school sessions to start gaining an idea where they are with respect to the work,
and then there is a growth. And so this sort of leads to further questions how is
that growth -- is that a syntropic growth and so on, so you can think of increasing
the number of sessions and so on.
What happens behaviorally? So this is what they say they perceive. So what
happens behaviorally is -- what we did is -- remember that I said in more than
half of the cases in the first study they were not aware of those changes, right?
So what we thought is in order to distribute better the people over the this not
aware of the prompt and awareness spontaneously is to increase the number of
prompts so that we will try to pull in some of those people that still were unaware
so increase the prompts up to three to see if we are still able to pull in some of
those people.
So what we found is spontaneously where we're more or less the same, about a
third, a third of the cases where people will just find the change, after three
prompts would be about half of the people were aware after the three prompts.
Yes.
>>: Would you say a word about what a prompt is?
GREGORIO CONVERTINO: Yes, very good.
So think of a situation where a change occurs within one tool and we just happen
to be in any other tool, right? So into the Groove case we just happen to be in
the foreground, we have the project manager and actually the change was in that
piece of report is an editor, cooperative editor. So what the confederate was was
always the same prompt. For example, in that case, if the two paragraphs of the
report has been inverted, for example, the confederate will actually bring
attention to that tool without referring to the paragraph, so we're just sort of
bringing it closer to the change, not refer to the change, but bring it closer, at
least shift the focus or the attention on the right tool, still see if he or she notice it,
then make a reference to bit closer to the change and see if he or she doesn't still
notice, and after the third prompt, if they don't notice, they are in those 20 percent
of the people. So you notice that even after three prompts there is still one fifth
of the changes not noticed which is pretty hard to how difficult it is to pull up
those changes.
>>: Do you think that the changes are equivalent sort of in magnitude, if you will,
across the different change types?
GREGORIO CONVERTINO: Can I respond that at the end? Because that's the
point of the discussion.
>>: Oh, okay.
GREGORIO CONVERTINO: Okay. So --
>>: Can I ask one more question while we're -GREGORIO CONVERTINO: Absolutely.
>>: You talked about the six clusters and the one (inaudible) fairly highly
(inaudible).
GREGORIO CONVERTINO: Correct.
>>: But you get this (inaudible). Were any of your other loadings more reliable
early on in session one and two, as you can imagine all that loading error is
driven by that second half.
GREGORIO CONVERTINO: Yeah, the answer is yes, but I think there were a
couple of factors that related. Communication is sort of unite the next study
basically. The common ground -- people have the perception of gaining a better
communication a bit earlier than awareness, that was, if you want, a very rough
response to what -- so one of those clusters was people's perception of the
efficiency of communication with each other, and so that would tend to increase a
bit -- it wouldn't be a big change anyway so it was a bit earlier than the rest, yes,
and that would show a bit more of a linear trend, if you want, rather than that step
function type of -- yeah. So did I answer your ->>: Yeah, yeah, I'm just trying to understand a little bit about why you did that
serious step function and wondering to what extent that depended upon the
specific (inaudible) so I can get a sense of what you're -GREGORIO CONVERTINO: Right. Consider the fact that of those factors I'm
just considering what is that 62 percent of the load.
>>: Yeah.
GREGORIO CONVERTINO: I'm just extracting that. So if that's really true that
that's what overlaps, I mean at least statistically, that's what you're doing in factor
analysis, you make assumption that what you're extracting is what's there in
common semantically.
So equal variables, the finding, the main finding here was that one specific skill of
people which is metacognition, which is people ability to monitor themselves,
how much are you able to see yourself, what you think, okay? Well, this ability
seems to clearly affect their ability to be aware. What really happen if you
actually see those -- their points with the -- if you split the people in high and low
metacognition people, right, so you will have the -- wherever the lower
metacognition people will end up after the fourth session will exactly be where
the other people will start so it makes a big difference. So if you can monitor
yourself, obviously you can monitor other people, certain metacognition, right?
So ->>: (Inaudible) the metacognition?
GREGORIO CONVERTINO: There is a standard, yeah, a standard scale.
>>: (Inaudible).
GREGORIO CONVERTINO: Standard scale, you ask, you know, and I mean it
is a control scale because it was generally used in business, I've seen it used in
business research as well. But basically you have some neutral factors that
people don't really -- can't figure out exactly what you are trying to measure, and
then you ask normally about their normal practice, how much they see and what
they do, if their planning practices are matching more or less those that -- of one
person would monitor it himself very well it's on.
But again, yes it's a perceived measure, yes. (Inaudible)
Systemwise, did we see any difference between the two systems? Well, the
answer is yes but if you ask the people the differences of their perception wasn't
different so what people were saying about their ability to be aware in the Groove
and Bridge system wasn't different. So we did not find any difference in the
subjective measures, but what they actually did was different. So into the -- into
the Groove system we had about 29 percent of cases of not aware against the
10 percent. So one of the system had clearly more cases of not aware even
after three prompts.
And also the question here is is it really true that those sessions -- that those
changes that are more of a style and more difficult to become aware of which is
the multi session, are really they more difficult, yes, I mean, at least has a
tendency again here it shows in 29 percent of the cases the multi session still
ended up into the nonaware category.
And so consequences, how about -- so if you see here over the graphs, you
know, just black bars basically were the nonaware responses so you see higher
in the Groove and the higher in the multi session scenarios. So whenever
changes occur between the session it is true that it's difficult to catch it.
Consequences where changes into awareness predictors of, you know, quality of
the work, well, in terms of subjective measure, yes. I mean, the more people felt
aware of their activity, the better was their perception of their products
perceptionwise.
With respect to external judgment, we had two judges assessing their reports.
We -- objectively we did not find quantitative difference between the quality of the
two reports. Qualitatively we only found that what came out of Bridge had the
more variability. So out of the one work space that had more of an integrational
across the tools, the product will actually end up have more relationship, more
logical relationship inside and more variable. So Groove was more of a
compartmentalized type of work space when you just move from one to the other
to the other. So the report tended to have the same shape somehow so sort of
the structure of the work space was priming the structure of the project
somehow, which is an interesting -- yeah.
>>: The multi and single are presumably averaged across Groove and Bridge
so -GREGORIO CONVERTINO: Yes, correct.
>>: Okay.
GREGORIO CONVERTINO: Correct.
>>: So there's kind of an interaction there so the report is a multi Groove -GREGORIO CONVERTINO: Yes, in fact, if I have to do a next study, that would
be exactly the question.
>>: You would have to do there, right?
GREGORIO CONVERTINO: Well, if -- I would like to have larger numbers of
cases. I cannot get very strong significance of interaction here those little one.
So I think this claim that those two have a possible interaction, I cannot claim the
interaction effect itself. So this would be an interesting -- so basically try to
contrast the most different work space I can, and then try to work on very
multi-session -- very single session type of changes.
So you can take off, you know, changes that occur in between the first and the
fourth session, I don't know, you know, things that will historically belong there.
So implication -- so what did we learn from those first two studies on awareness,
well, the goal here is try to -- we know the last critical changes and that they will
trigger confusion and people will just -- going to break down, will need to recover
and so on, and so all this overhead, we want to provide effective enough
distinction of those changes if we know that they are critical changes and so back
to what do you do about -- I think was your question, what you do about, you
know, the visibility of changes.
The goal here is try to model what are really the parameters conditioning
people's ability to be aware of that event. And what we learn from here are that
some characteristic of the event as we saw, the temporality, the fact they occur
within session, between session, that affects people's ability to be aware of that
event so temporality of the event is important.
The granilarity (phonetic) of it so if the change is really an entire document
versus a paragraph of a good document they may not fare as well, and if it's
concrete or not so it can be an abstract change, and what happens with last
change is for the last change was priming you to expect a similar change as well.
So historical relationship between the changes are also important. So those are
event characteristics if you would.
Then you have user characteristics. So we sorted the ability -- people's ability to
monitor themselves that affect their ability to monitor other people for example,
and finally work space characteristics. So if you have a work space like Groove
that is constantly imposing you to jump from one work space to the other. So
here is an example. One basic observation if you want in the construction of
work space, which is an implication for the sine is we always assume that people
plan and then execute, right, which is never the case, obviously. The problem is
that if you structure the work space so there is a communication between your
planning tool and your actual execution tools, what I am constantly losing is what
did I say I was supposed to do and where am I currently with respect to that.
So people -- first of all, people don't have the plan. That's why you have the
delayed effect, constantly they will sit there overestimating the time they had and
at the end they will realize they didn't have enough time, and had we planned in
time sort of type of comments.
So the point is in the other tool there was a versioning system so, yeah, I could
see the version of my documents and then I could see the relationship with
respect to the deadline. So if I see the deadline and I see a bunch of versioning
just before the deadline, I more or less see that there is a level of activity there.
The other tool I either have to plan or I execute. I don't see the relationship
between the two which is a critical aspect for making people aware especially in
the long-term type of work.
Finally what did we learn about the awareness process, we already spoke about
this warmup effect on the group. There some control variables that you need to
control, you may think that you are having an impact on the user, but they just
end up to have higher metacognitive skills and you just don't know it so you
better control for those.
So again the last point here is that we did see different stories based on
subjective and objective measures. So what people think they are aware of is
not exactly the same story of checking their behavior so you better have a couple
of those measures together just to see -- those discrepancies actually are more
important than the concordance of those two measures just because you exactly
want to know when people think they are aware but they are not actually.
All right. So second pair of studies that I'm going to present quickly, I hope I'm
still in time, I know it's late, okay, so from awareness now we are moving into
more narrower area of investigation so you can check more about, you know,
how those activity awareness referred to common ground, this is a framework
paper that I wrote with Carroll and Rosson and is interacting with the computers
in 2006 that sort of framing how activity awareness -- sort of -- in other words,
we're talking about people's ability to coordinate activity in the long term, right?
So a basic function here is they are able to share knowledge. So a basic
primitive function before we start talking about are they able to coordinate, are
they able to share knowledge, right? So sort of the basic primitive function there.
So we decided to start studying more in detail that specific function of groups,
okay? So that's why we focused on common ground and grounding.
So we went back, we looked at the literature, what have people already said
about common ground, and tried to transfer those -- this existing theories and
methods into team work.
So why would this be relevant? Well, if people are able to share knowledge, I
mean, if we really want them to work efficiently, you do want to facilitate that
primitive function, okay?
So we -- the problem is that we know a lot about common ground in CMC and
face-to-face communication. There is really a lot of communications, but we
really don't know much -- as soon as you move this function into the context of
work we don't really know much about it. So what are the interaction with respect
to higher level processes such you're making of production, right? In addition to
the information transfer.
So the question here is can we measure it in this new context. If we can
measure it and model it, perhaps we will have insight about how to better support
it, okay?
So a basic distinction, so if you take the literature on CMC and Clark, Clark's
work and Monck (phonetic) and later on that have applied this to CMC and
compare it to work, what are similar aspects? Well, it's still true that
communication in the context of work is still a collective process which is the
main basically brain shift that Clark brought in from the information theory type of
view of communication, right? It's not just the sender and receiver, but it's a
constant checking and a joint action between two people, right? So it's still true
that it's a collective and adaptive phenomenon, adapt in the sense that the
principal minimum effort is really what is driving people in communication that will
use the cues and the channels that will actually allow them to put in the least
effort, right? So the best end is multi model.
Now, the multi model aspect becomes particularly important for us because we
are talking about modulating artifacts so the number for example is a very
important aspect now especially if you are talking about work.
Now what is really different with respect to the concept of Clark and followers is
that as I said before, we are talking about communication but it's not just about
communication. What we are really interested in is the higher level process. So
what you can really think, if you take the experiments of Clark and collaborators,
are very elegant yet they are basically an information transfer problem, right? So
there's one leader and one follower and they have to make sure that information
gets transferred from the leader to the follower in the most efficient way. So they
keep doing it and over -- over time they see that their ability, their efficiency in
doing this transfer becomes faster and faster, right?
Now, the problem is that in the context of teamwork, people as they start the task
they don't have the hundred percent of the information they will actually have at
the end, right? So in the case of the leader and the follower the information is
already there on the desk of the leader, right?
In the case of CSCW groups, they have to generate how to go about doing the
task in addition to transferring information. So there is a generation of
information in addition to transfer, that's all I'm saying, the actors were moving
towards teams and not just there, and more importantly the information is not I
know that you know that I know that, it's also that, yes, is content, but I know that
you know how. So it's really the how is the procedural, is the strategic thing.
So in Clark's example, the classic example he does of the grocery store
assistant, so I go in the grocery, the script between me and the grocery store
assistant is based on a set of assumptions. We really need to say very little
things. I go and get the express, I just have to do, you know, do a thing -- single
turn of communication with several -- you know, several implied ones.
The problem of work is that there is not -- sometimes the way of going about
doing a complex task is not already defined, it's not the social script that we
already know. Actually, a part of our job is to the define the strategy especially
for experts, or sometimes we need to define task, what the really -- the big
contribution they give is to find the best strategy to attack the problem. So when
they start, they don't know it yet. So the know-how the information is really what
we are after here.
So we study common ground in a face-to-face setting as we start so they are
collaborating on emergency management task, I'll tell you more, so basically we
are modeling teams of experts where there are different experts within the same
team and they are trying to plan something over the maps, and what we try to do
out of this -- this is just a basic overview, this would be enough for the rest of the
talk, but basically we try to measure aspects of this process, how do they
increase in their ability to share content and process common ground. We try to
compare across different media so we have a study in face-to-face on paper
prototype and then we did exactly the same task in the same content they did it
over a softer in a distributed fashion, and we can compare the same measures,
and the final outcome here is that we want to learn better ways to support those
teams, especially in the more difficult situation like the distributed one. So what
additional tool would you introduce so that they get better into the incremental
process common ground.
So those are the two studies just briefly outlined in terms of methods. We have
three-member teams using paper and face-to-face and three-member teams
using computer support in a distributed fashion. What we change, what we
manipulate in this case is the amount of shared knowledge so we have the same
type of task with different emphasis. They're always planning over maps, we just
change the content and the distribution of information over the map, and we want
to see how better do they get with that type of task over time. We have a half of
the teams prebriefed about each other roles and half of the teams just know their
own roles so we want to see if we can artificially increment the amount of shared
knowledge as they start the task, okay? As they start the set of runs, and finally
we try to compare the medium. So we measure process and outcome, and here
are some results. So three main orders of research question you want to ask
here is this, do those teams increase in their ability of, you know, sharing
information. So does common ground increase, yes or no. Does it increase
because of the run so the amount of shared knowledge, which is one aspect that
we communicate, does it increase when we give them additional information
about each other roles, does that change the way they interact, and, finally,
what's the effect of media. So those are the basic questions.
So first of all, we need to know if there is an actual increment. Then, if there is
that increment, then we can ask how, how did they increase it. So how was the
interaction changing in the context of a complex task, and I'll show you a couple
of results from their verbal interaction and later on you want to extract
implications for research and design.
My computer is a little slow.
Well, I can tell you more about the conception model later if there are questions,
but basically this is the task. So there's an emergency manager and planning
task. Now, an important thing I want to say here this is a model of the task of -coming out of real emergency management team, so there is a colleague that
has studied emergency management teams in Pennsylvania. There's actually a
paper in the journal of CSCW on this I think this year. She, Wendy Shaffer
(phonetic).
We found that those teams that would do local emergency management will
actually have a bunch of different experts and they will meet about once or twice
a year do those running of typical scenario that will need to be prepared in case
there is an emergency. So we have those tabletop exercises.
So building on that and on FEMA description of roles, we decided, okay, so let's
have a realistic model of how this thing will do this tabletop exercise. So we split
into three teams based on FEMA roles, so there's an engineer who is public
works expert, he knows about roads and bridges and utilities. There's an
environmental guy, he's a geologist, he knows about the rain and weather
patterns, and there is an allogistics (phonetic) person, he knows about
transportation so we just splitting the expertise across those three people in a
realistic fashion, and each member has his or her own map with role-specific
information on that specific map, a list of risks that he or she is going to be aware
of given that expertise, and then a shared map on which they have to plan the
rescue mission. So what we are supposed to do is to build the best plan for
rescuing a family from a flooded area so this family is right in the center of the
map and they have to decide to bring it to one of the shelter that on the
periphery. So every time we change the difference of the task, we just rotate the
four shelters so that we become independent of the locations exactly. So they
have to decide what's the best shelter and then rank the risk, okay, in terms of -now, the trick here is that we try to study these in order to -- what we really want
to know is how well did they share, right? So we want to make the quality of the
product dependent on how well they shared. So we use a profile that comes out
of group psychology, and they use it in a flip way so we sort of flip -- so here the
solution -- the people are the columns, so the public works, the environmental
and the MC here are the three experts, and the solutions are A, B, C and D,
right?
So what they really receive -- so remember that rolewise what wins of the
solution is the one that has the least number of risks, okay? And we sort of
pretested those risks before to make sure that they more or less had the same
weights, okay?
So if they are thinking about their role information, which is what they have in
terms of columns, they will try to anchor to a suboptimal solution. So the public
works is going to anchor on A because he thinks there is only one risk, B and C
and so on. So they will try to anchor on A, B and C, but only if they immediately
share together, they understand that there is four risks versus seven of the
others.
So it's a big effort, and you systematically see from the literature a bias in groups
that people will constantly keep anchoring on whatever they thought. So this is
very strong to see. If they really share efficiently, they should see that visible
difference popping up, right?
So you may have different problems here. You may have too strong of a leader
that tries to pull too much and so on. So there may be several reasons why they
fail, but in any case the quality of the sharing is predicting -- is somehow affecting
their ability to get to the optimal solution, okay?
So what did we do? We did preliminary analysis of the media and qualitative -quantitative analysis then of questionnaire, verbal interaction, there is a lot of
work we've done on verbal interaction in terms of current structure with Sellen's
(phonetic) measure and content analysis. So I'm just going to present a couple
of this, just as an example, we did a recall measure after the task to make sure
that we could measure how much they would retain of the what they did, and
performance measures. We're currently working on the number, we have some
preliminary data if you want to (inaudible).
So we say the first question is did common ground increase. Well, the answer is
yes, as you would expect, it did increase over time. The point is here what -what really sustains that claim, okay? So it's not -- first of all, people proceeded,
you know, that difference is quite significant, people perceive that as they repeat
the same type of task with different content, they do perceive that they have, you
know, constantly greater sharing and greater sharing about the how as well,
okay?
Now, if you -- just X-ray their communication and you sort of see their
communication as segments over a timeline, right, so as you see this. So these
were measures developed the first time by Sellen and Park (phonetic) in '92, '95,
when they were assigned CMC type of tasks.
So they will see basically that those segments of communication, the temporality
of those segments will change, and also the overlapping of those terms will also
change. So we transcribe their communication and we also saw the starting and
the end of each run. So you will basically see that by the third run, they will tend
to have much more compact turns of communication, shorter, and the interesting
thing is that if you were keeping track of where they would overlap, for those
overlapping of turns that would not being interrupted, so when I'm trying to
overlap with you and grabbing the floor, those types of overlap would not change
significantly, but those that are failure of grabbing the floor, those would
significantly be less.
Now, to be clear and fair, those could be also an explicit acknowledgment, you
are saying something and I say yes while you are saying something right?
>>: Could you explain what's on the vertical axis (inaudible).
GREGORIO CONVERTINO: Those are the three role, the three people.
>>: That's what I thought.
GREGORIO CONVERTINO: The three people. So you see basically 19 is an
individual turn of that specific member, right? So just an example. So you see
the individual turn from Run 1 to Run 3 becomes significantly shorter so ->>: Not it's not the TW is always last.
GREGORIO CONVERTINO: No, no, no, no, I'm sorry, I'm sorry, I'm sorry.
Yeah, this is just pulling in -- and (inaudible) so the thing is that they were able to
say the same content, actually generate better solution in a shorter type of
message, and, yes, that is basically the overlapping segment so that overlapping
segments would not be missing there.
So we also measured the recall so what we did is we knew what information they
had as critical pieces of information as they started so PW and AN and MC again
are three roles, right? So we could keep track, we asked them to rate as base
factors individually what they thought was relevant, right? So as they went into
the discussion, we then coded the discussion to check whenever one of those
pieces will surface, the group discussion, and after the task when we asked them
what did you think were the main issue discussed that were leading you towards
a decision, if that piece of information was still coming up as a recall and would
check with each other, well, then that will also be an objective measure of
common ground. We did really ground on that type of content.
Now what we could see is from -- between Run 1 and Run 3, the ability to recall
relevant pieces of information after the task grew significantly from what
(inaudible).
Now, the lesson here is if it -- it goes a bit beyond because this content common
ground so obviously people would say, well, people will learn. Well, the problem
is that in addition to the fact that they have to establish the strategy, okay, so
they have to allocate a certain amount of energy in establishing the strategy,
right? And then later on assuming the strategy once it's established before you
have let's override on the (inaudible), before you can focus more on the content
so this is not just what you see as a face study of this content, but it's also due to
the fact that they -- at that point they can release the part of the how, okay, which
is what we also notice -- which we -- I mean, I can claim that because it came out
of the content analysis, okay? I can show you about that.
So as we see here, the -- they can do the same task in a significantly less time,
and the trend is the same, so we did link the performance to the ability to share
and this really through the -- over time they tend to produce in a shorter amount
of time, so basically the outcome becomes better in terms perceived measure, in
terms of completion time, and the softer -- the surprising aspect is the softer
groups actually tend to start producing better from the beginning, probably
because the tool is focusing them right on the pieces of information and they
have less distraction on the social level. We don't exactly know why that's the
case, but the softer groups tends to start producing optimally better solutions
from the beginning, okay? Which is an encouraging thing. I mean at least we
can -- in terms of -- so generally in the literature you have that the CS guys are
always the -- you know, the guys that lose. So this is surprising in this sense.
So what did we see in terms of the how? Okay, fine, we have established that
they do increase in their ability of common ground. Now, remember, it's not just
what they say, it's also the structure of their communication that changed, okay?
It's not just the learning effect, it wasn't just a constant learning would not be a
strategy of communication as well, and there is also recall. By the end, we
checked that they did recall more. So once we have established that they do
increase, we want to know what's the mechanism behind, what was happening in
terms of function of messages of people. So we categorize them in terms of, you
know, acts that will serve the checking on an understanding, acts that would
serve the transfer of information, information that is being -- is reaching the table,
and then managing the process and making the decision.
Now, I'll just show a couple of examples of what was we notice by doing content
analysis. So consider the fact that we are -- if we are a team, we are supposed
to share information as soon as possible, information can reach the table either
by you asking me about it and me relying, which is the rules, or simply later on by
me pushing it directly because I know you want it, right? So that's much more
efficient, correct? So that's why we have -- we try to distinguish the idea for -- the
idea for a type of move from the query and reply, okay? So we want to see if
that's making any difference.
We also -- you know, coming out of the literature from Clark and collaborators
you want to check if really the checking clarify is going to really disappear. They
would predict from their experiments that the more people get accustomed to a
task, the less they need to check and clarify, okay? That's what we would
expect, those moves will actually go down, right?
And finally we want to see how much investment is given at the level of
management and judgment. So this is what we found. So from the face-to-face
which is our baseline case, okay, you don't have any constraints, people can talk
to each other how quickly they want and so on so we did see that the pull which
is me asking and you giving it to me, that's the, you know, the two-move strategy,
is decreasing where the push, me pushing directly is increasing. So that's the
strategy information transfer is actually changing so that's a bit of a how we get
better in sharing.
>>: Is this (inaudible) number 5 percent 10 percent?
GREGORIO CONVERTINO: The percent all the moves that we do, the push will
have that relative amount with respect to the total number of moves, yes.
Frequency, total -- yeah, frequency, yes.
So this is -- this is we compare high performance and low performance of the
team so we just rank them by performance and you see that high performers
tend to clearly increase in their push. But basically the pattern is identical so
people get better in terms of -- so here you see -- see if I'm able to do this.
>>: I believe (inaudible).
GREGORIO CONVERTINO: Push.
>>: (Inaudible).
GREGORIO CONVERTINO: Push.
>>: (Inaudible).
GREGORIO CONVERTINO: No one is asking her, she's just pushing at this
point. It's just an example of how they move from the pull to the push.
>>: (Inaudible).
GREGORIO CONVERTINO: Okay.
>>: You say you recruited the participants, who the participants were.
GREGORIO CONVERTINO: University student, yes, they were all university
students.
So are the examples. So in general, if we compare the paper in the softer study,
you see that the push and pull is clearly the same more or less, and that's a bit
more evident into the softer study.
In terms of check for understanding, this is the interesting fact, especially in the
paper study. As I say, you would expect that they do decrease in the moves that
are checking. As they increase common ground, they hit the ceiling, we don't
need to clarify anything else, so therefore we lowered those moves. What really
happens is that there's no ceiling here, I mean, we are just increasing the
complexity of the task, they have the perception they have more information -deliberately we give them more information than what they can handle, and you
see that they more aggressively keep checking and clarifying with each other just
to cover better the information space. So this is a clear example how C type of
prediction do not apply to teamwork type of setting.
If you give people a decision-making task that has a complexity that has not a
clear boundary as in a communication task, those type of moves will keep
increasing. More interestingly, those that are overcompensated are exactly
those that have difficulty with the task which are the low performers so you see
that trend of checking of the dotted line there are the low performing teams. So
the more I struggle with the task, the more I keep checking and clarifying over it.
So that may be a nice indicator to see when a group does really need some
support for common ground.
Management acts, this is basically the claim for the -- that increment is actually
significant. The more people go ahead, the less they need to say how we go
about doing the task, okay? So by the end we already know. That's why I said I
can then allocate that energy on the content sharing, okay? And in the softer is
less clear, this is less clear, especially for the high performing -- I'm sorry, for the
low performing. The low performing completely -- in this situation is completely
unpredictable why they are increasing their management acts, probably
overcompensation as well, I don't know.
Finally, the one aspect of this unique of softer groups, and this is probably the
most important point for implication for the sine is that over all we found that the
softer groups would perform better in terms of information sharing, just totally
because they were focusing on those pieces of information, they would just
efficiently get the job done and sharing it. But the second part of the story is that
once you start looking at their judgment acts, they tended to make more explicit
their judgments just because probably they could not see each other and so on,
so the lack of views will -- rather than having the decreasing trend in the
face-to-face study where they will feel less and less the need of making explicit
their judgment, in the case of softer study, they are actually making more and
more. So maybe in that situation the lesson we learn out of this is that -- I'll not
confuse with you the images here, the lesson we learn is that on one hand we
have a tool that is sufficiently supporting information transfer, okay, they're doing
a good job. On the other hand, they do judgment or decision-making part of the
task that is not yet well supported, and I cannot -- I cannot easily refer back to my
prior judgment, I don't see it, it's ephemeral, right? It's more ephemeral than the
face-to-face situation so perhaps that's an indication of where CSCW type of
tools for those types of tasks should concentrate their attention to, right? So
more on supporting the judgment aspect of it. How do you make them more
visible so I don't need to make it explicit, right? That's an added cost somehow,
right?
>>: Is it bad to make the judgment explicit?
GREGORIO CONVERTINO: Well, you need to make an extra move basically,
right, as opposed to the situation where we can just simply assume it, right?
>>: Yeah.
GREGORIO CONVERTINO: So this was what the tool looks like, there is a role
map, there is details on the role, you see how crowded is their map on the left
and on the right is where they build, they have telepointers over here where they
see each other pointing to stuff, like I can roads and so on, and this is more or
less the architecture, we have the server data which is the collaborative
architecture, and we have both shared map data and shared awareness data
which is basically the infraction level of the people like them selecting.
I didn't point out the fact there is basically color coding. I can know who is
selecting what and what traces are coming from whom and so on. And so
basically what we are working on is at the level of the futures level, so annotation
tools and so on, and so where we are going with this just generally is on one
hand trying to generic process organization, some aspect of their process, so for
example, I show people how much contribution they're making over the map,
maybe if someone, for example, is doing not enough contribution, they can
readjust, for example, or can I help them with telling them how many cons are
there by shelter, by area, so they can start, you know, understanding what's the
bonds between its possible solution and so on. So this is the direction where we
are going trying to understand how to better visualize information back to the
group so that they improve and annotation tools as well.
>>: The process visualization, it makes sense to me in a scenario where equal
contributions of all participants is the optimal thing, but what if you're in a
situation where there really should be one leader and a few listeners who aren't
participating as much, do you want this to encourage people to constantly chime
in even if that's not optimal for the situation?
GREGORIO CONVERTINO: Right, that's why we picked the emergency
management expert themes as our application domain saying those things you
do want -- you do want to make sure that each one of the team expert has a
voice in it because they're supposed to be the type of team experts that you bring
in as a team and in half an hour they have to bring up a solution.
>>: Right. But I was trying to generalize -GREGORIO CONVERTINO: Right. I think you -- you're perfectly right, it's really
task type dependent. Yes, absolutely.
But let me get back to that. So think of a situation, this is actually the scenario
where I'm talking -- thinking about in the future, think about a situation there was
a case of a BBC article six months, eight months ago, last summer basically, I
don't know how many months, so there was a fire in Greece. I don't know if you
remember, there were several people dead in Greece, so there was this BBC
article which was fascinating, especially in making the case for research. So
they had -- basically what happens is the fire is up in Greece, soon, relatively
soon they realize that Greece emergency management could not handle that fire,
it's too big. So the demand goes directed to the European, U level and there is a
control center in Belgium, and so at that point the guys over there in Belgium
have to coordinate the operation of rescuees and people rescuing and somehow
handling the fire in Greece.
So you can have a situation where you have the field exert that is right there in
the field, you know, with a handheld, for example, and he has to come in here
with those guys that have very large screen displays for example. Now, the
problem is if you really want to ensure that that guy has enough voice into the
task, you will try start imposing the problem of this, how do you compensate, for
example, the lack of real estate on the screen of that guy, given that you know
his expertise is as valuable as the other guys, if not more important. So this is
where this is trying to go. First get a model where you can claim that they have
to contribute, I don't know, 30, 40, whatever, and then say how can your
visualization help compensate possible constraints that they have to work on.
I think I am done. Yeah. So basically, generally speaking both threads of study
are trying to bring towards a situation where you can improve more the long-term
working sense of reduction of overhead. I spoke about this in terms of
awareness and also in terms of common ground and methodologically, one of my
efforts, especially for my Ph.D. dissertation was to try to create methods that
specifically address the needs of CSCW researchers. So that you -- you should
tune those methods rather than simply importing methods from psychology, now
there is not necessarily knowing if they're really serving our purposes. So it's
trying to move a bit from protoscience where we are now to some more scientific
approach, from protoengineering to more (inaudible), and this is where I'm going
so I think it was clear from visualization, and, finally, one last aspect is trying to
look into tools that can increase professional development. So when people -actually by working together, they actually are trying to gain better skills so think
of a group that helps better to self-regulate once they have understood the type
of bias they're going to have so can you prime them and help them to interiorize
those strategies later on. That's all. Thank you.