>> Jaime Teevan: All right, so we're going to get started. Welcome. I'm pleased to welcome
Lisa Yu today. She did he doctorate at Stevens Institute of Technology, where she worked on
suing genetic algorithms to understand creativity and support distributed cognition with a crowd.
Since then, she went to work with Niki Kittur and Bob Kraut at CMU. She did a two-year
postdoc there, looking again at creativity and distributed cognition across groups of people,
looking at using analogy and divergent thinking to help people innovate and do new things,
things that you maybe think that you could not do on your own, so we're very happy to have you,
Lisa.
>> Lixiu Yu: Okay, thank you, Jaime. Hello. Hello, whoever is out there. So my name is Lisa.
So, actually, I want to start a little bit with my own history. I did my PhD in management
information systems, but I did a lot of projects in psychology, cognitive science in particular. So
back then, we were interested in the role of technology in the individual creativity -- that is, like
how can we study what's happening here, and now how can we develop technology to support
people to solve problems creatively or do tasks better? But then when we did all the projects
trying to design technology, we find that in most case, actually, when technology like computers,
mobile phones, are involved, it's no longer confined in the single mind. That is, it's distributed
across a small group of people or a large group of people. So this idea is called distributed
cognition, actually has been around for a while. Originally, it was coined in the late 20s, and so
the basic idea is that cognition or cognitive process no longer happened in a single mind or
confined by whatever you have here. It actually can be distributed to a lot of people. Especially
nowadays, we have the Internet. We have all these web technologies, so it is possible to involve
as many people as you want through technology. So one example we're all familiar with is opensource software development, so in the past, you might have a group of people within a company
who want to build something together. But now, in the new environment, you can involve all
these strangers, all these programs that you divide a project into pieces of work and the people
build on each other's goals. And another popular example is Wikipedia, so people start to
accumulate knowledge in a shared space. So a large group of researchers start to study in these
areas, and we have some answers in how can we distribute this kind of task to many people, how
can we allow them to build on each other's work, so we know something in this area. But there's
a third area we know less, and there's less study in this area. It's called crowd design. So a
website like Innocentive, Threadless, Quirky or 99 designs, so in these websites, what people do
are slightly different, meaning they are here to do some kind of design, like local design or
design a tee-shirt, or something where people actually try to solve some scientific problem that
cannot be solved in a company. And in Quirky, which is a new company, it's even more
interesting. It pushes even further to the crowd, like people can find whatever ideas they think is
interesting, and add their ideas to the website, and all these people help them to select the ideas,
develop into products. Then, they sell the products on the websites. So everything is conducted
by these people. But on one hand, all these tasks, they are hard, right? Design something novel,
do this kind of a creative task, it is very hard. We kind of need all these minds together to do
something great. But, on the other hand, this task involves very high uncertainty. Even in the
traditional psychology, we are not sure what's really happening in the mind if you try to solve
problems. It's not some article you can break into paragraphs and sentences, then people just
recount these small things. How can you really look into the thought process? How can you
control certain parameters to make it creative? Suddenly, you have this great idea, you design
computers. This kind of thing is very uncertain. So that's why it's hard. That's why I am
personally super-interested in this area. So from a practical point of view, I'm interested in how
can we help people to do more innovative and complex work than they can do on their own. So
this kind of work can range from small, innovative products, consumer products -- can be used
every day -- to very high-technology devices or even to more fancy, like complex data, like
buildings. So by studying this phenomena happening in this work now, I fundamentally
theoretically am trying to answer these questions, how can the cognitive process we normally
associate with an individual mind can be implemented in a large group of people, mediated by
technology? So in the rest of the talk, I'm going to talk about two projects I have been working
on, trying to answer this grand question. And then, at the end of it, I'm going to talk about some
exciting future research. Okay, so I started from asking the question, what's going on in a single
mind? Like, when people are trying to solve problems, trying to design something, what's the
process of this creative thinking? And there are a lot of literature in psychology, and there's
people trying to study what the creativity sparks from. And one of the explanations is called the
combinatorial creativity. So it argues that, actually, it's seldom that ideas come from nowhere.
Most of the time, they are built on the previous ideas. So if you have all these previous ideas,
actually new ideas are based on the combination of the creative element of the existing ideas, or
you put the previous elements of the ideas in a creative way. So this is what's going on in the
individual mind. Then, what's happening if you push it further? Do you distribute it to a lot of
people? As humans, we have built this culture for thousands of years. How did that happen? So
there's a word called memetics, so people started memes, so they treat a unique culture like ideas
or certain kind of technology. You need a culture, like they are passed on generation by
generation to different people, so the central process of force in the culture propagation is
selection. Other people have ideas, and the third person, the second person might pick on these
ideas and use it in a different way. So people might have different ideas in this line of work, but
the basic idea is selection plays a very big role in cultural transfer. So there's combination and
selection, so we're trying to build a process that can incorporate these two forces, which I will
call the evolutionary process. So this is how it works. Suppose you have a design problem, so
instead of trying to solve or coming up with ideas yourself, you can post it up there to get access
to as many people as you want as a first generation. They can have solutions for you. Then, you
evaluate them, select the good ones and match them together into pairs. You present a pair to a
second person and ask this person to generate new ideas by combining them. And then, you
have this feed of ideas. You get a second group of people to build on these ideas, have new
ideas. You run this process again and again, so this is essentially evolution in process, like two
people getting married have kids, and hopefully kids can take the best genes from the parents
that they become more nice or smart or intelligent. So we are hoping through this evolutionary
process, driven by combination and selection, designs can evolve toward a better, higher quality.
So then, we need people. We need people to actually carry out this process, and luckily we've
got Amazon Mechanical Turk, so we can access as many people as we want, so that's how we
get participants. These people have no design background at all. They are like some of them are
stay-home moms, some of them work in construction, or they're distributed around the world.
They work in different professions. And we direct these people to this design interface, where
they can text, write their ideas, or they can also draw to realize their ideas. So this is what really
happened in the process. So all these ideas were the ones produced by this crowd of workers,
and so this particular design task was designing a chair for children. So that's why you can say
these are [indiscernible] features, so they need to consider the material, the attraction to the kid,
all of these things. So you have the first generation where here they get it being passed on,
combined into the second generation and the third generation. So the first question we want
answered is do designs evolve later better, so we compared the designs in the last generation
with the ones in the first generation, and we also want to see whether this process produced
better ideas than alternative processes. So this alternative process was independent idea
generation. We basically mimic what happens in the real world, like people generate their ideas
on their own. So then we need to figure out a way to measure our results, and so in psychology,
for the people who study creativity and designs, they basically argue we need to consider two
dimensions when we judge the quality of ideas. The ideas need to be novel and need to be
practical. If it satisfies these two dimensions, then you can read it in some way and converge
these text and images into numeric values and then compared the quality of ideas. So that's what
we did. That's how we judged the quality of ideas.
>>: When you say high inter-rater reliability, was it on previous things that they've done before?
>> Lixiu Yu: Oh, this slide shows how we actually rated the ideas, so the specific method.
>>: So the judges actually did have high inter-rater reliability.
>> Lixiu Yu: Yes, yes.
>>: So in the design for the multiple generations, your example versus the one before that, if
you go back to the slides for a second, so here you show something skipping from the first to the
third generation. In the previous one, I thought you'd omit them, so how do you limit the types
of combination that happen in the process?
>> Lixiu Yu: You mean how ->>: This slide goes from the first to the third generation. How are people actually constrained in
what is their starting design and how they can evolve it? Because in the previous slide, you
showed just limited, where each layer can only depend on a previous layer, so I'm trying to
understand how complex the policy is overall.
>> Lixiu Yu: So when we actually implement this process, we adapted genetic algorithms, so in
those algorithms, it has different ways to do this. One is called mutation. It's like you have this
idea you change in some way and then move on to the next generation, and they have this
particular process called elitism, like you keep some ideas which are rated very higher quality,
like they are very good. You move them onto the next generation automatically. And the third
process is automation, so you would randomly pair two ideas, but it's more complex than that.
You would find the two ideas, pair them together, find the better ones, and then you find another
pair, you find the better ones, and then you match them up, give them to a person and ask them
to generate new ideas. So this kind of process, kind of favored ideas that would be of higher
quality, so this process, basically, it has combinations, but it tries to use algorithms to favor the
ideas with the higher quality by either increasing their chance of being selected or just moving
them directly to the next generation. That's why this idea never -- first, this example, and even
though it didn't get combined here, there is a lot more data combined with other ideas. Second is
some ideas is moved because it is considered very good. They are moved directly to the next
generation.
>>: So one of the problems about genetic algorithms is you need to break your design into
several of the genes, and how do you actually break a design into genes? Kind of to me it's a
whole package. It's really difficult to break into several elements and select from them.
>> Lixiu Yu: Why we think is great about this research, in traditional genetic algorithms, they
can solve the problems which can be broken into genes, but design problems is very hard, so as
designers, we don't really know what kind of features they should pick and how can we divide
this design task into multiple ones. So we basically rely on the crowd to pick the features to
decide which genes they should pick, they should use, to how they can generate new genes. So
what we did is just designed, plan-formed this process, and whatever happened is in the control
of the participants. So later, we had a paper analyzing how the crowd workers picked the genes,
the features and how they actually do the combination. So the basic finding is they favor the
parts or the ideas that are practical and novel.
>>: Just a second question, so you probably mentioned this. So how is the evaluation of each
generation happens? Is it by some designer or it's by the cloud?
>> Lixiu Yu: Yes, we did two different ways of evaluating them. First is we find two designers
in our architecture design school, asked them to read the ideas based on two dimensions, and
then we also posted all the ideas on Amazon Mechanical Turk to get I think 2,000 Turkers to
read all these ideas and calculate whether the ordinary people, the crowd workers, think
consistently with the designers. And we got very high consistency, meaning that the -- because
the design task we used is some kind of product everybody understands, like chairs for children,
so we can try two ways of reading the ideas.
>>: Were they implemented in each kind of user or its own homogeneous way? Like, do you
weight some users higher against others?
>> Lixiu Yu: No, we just treat them homogenously. Okay, so let's look at the results. So, first,
we want to know whether we get better ideas through this evolutionary process, and so we tried
multiple ways to measure the results, so one way is to count the ideas, like if the idea is -- the
two dimension is a hair above the average scores of our ideas, then it's classified as a creative
idea, so there's a binary classification. And we want to know whether the number of good ideas
increased through this process, and that's what this figure shows. In the last generation, we got
much more creative ideas than the first generation. We also used other ways to count the
average quality, to see whether on average the ideas improved. We also got significant
difference across generations. So this figure shows more detail about how ideas evolve, so that
the white dots represent ideas in the last generation. The blue circles represent ideas in the first
generation, and so the two axes represent the originality and practicality, two dimensions, and
you can see that ideas collectively shifted to that corner, meaning they improved on both
dimensions. So we also compared the ideas from this process to the ideas generated by a
greenfield system, mainly independent idea generation, to see whether it produced better ideas.
So it shows through this kind of a process, people are much more likely to general better ideas
than they just do on their own.
>>: How does the greenfield system work?
>> Lixiu Yu: Just independent. Like the same design problems, and ask the same amount, same
number of people to generate ideas on their own.
>>: But that condition, the number of people in the first layer plus the number of people in the
third layer?
>> Lixiu Yu: Yes, that's a good question. Yes, so we ran for three generations. I don't
remember the exact number we got. I assume we got 500 participants. So in the control
condition, we got 500 participants, as well. Actually, we also controlled the timing, because the
generations happens over time, so we actually divided those 500 persons into the 300, so the first
generation, we run 100 at the same time. The second independent, another 100, to try to control
the timing, so everything trying to happen in the same time.
>>: So what is the total number of laborers that are in each condition? The other is, and I think
you're referring to the time on task.
>> Lixiu Yu: Yes.
>>: In the case that a given design that makes it to a third layer has had N units plus N units plus
N units of time, whereas a single pass by even a larger number might only have only the N units
of work spent on it. And so you encouraged -- you're saying in the greenfield system, you
encouraged people to work three times as long on the task?
>> Lixiu Yu: Oh, the timing? So in the evolutionary process, we ran it -- so there's time lag.
We run it, so suppose we run first generation the first day, second generation the second day or
second hour, then we run simultaneously, like get a same number of people to do independent
idea generation at the same time. But in terms of how long they spend on the task, we don't have
control over that, because even in the evolutionary process, some people, they spend like a whole
hour trying to make it perfect. We actually pay them $0.05 back then in 2011. So we cannot cut
it off, like you have to stop at this time.
>>: But in the analysis ->> Lixiu Yu: Yes, to see whether people spend a longer time there?
>>: Yes, whether that results in greater quality compared to the relative total time spent on a
task?
>> Lixiu Yu: Yes, when we did the analysis, I controlled a lot of variables, including education,
design background. It happened like in 2010. I don't even remember -- I think we controlled it,
but I need to double check.
>>: [Indiscernible] the coordination of quality as to design, because it might not actually matter.
The second, fresh set of eyes might make a huge amount more difference than the same person
spending more time, so it would just be interesting to see.
>> Lixiu Yu: That's a great comment.
>>: You mentioned quality earlier, but then you're presenting graphs on creativity? What
measure of creativity is there?
>> Lixiu Yu: So here, we basically equaled quality with creativity. Because we study creativity,
we care more -- we cannot really put these ideas into manufacture and sell it, so we focus on the
prototyping steps, like ideation, so when we talk about quality, we actually mean creativity. So it
means it needs to be novel and we need to practical.
>>: As rated by these two people with high independent rater reliability.
>> Lixiu Yu: Yes.
>>: So there's some question as to whether or not they can identify creativity or what creativity
means?
>> Lixiu Yu: Sorry?
>>: So, like, often you can go to somebody who for example doesn't know the related work well
and ask them, is this novel? And they say, yes, that's novel. And you show it to 30 other people,
and they go, no, of course, that's not novel. This has been done here, here. So I'm trying to
understand, what in how you're measuring it with just two people, can two people identify
creativity? Has anybody actually looked at this concept of what it means to identify creativity?
>> Lixiu Yu: Yes, so that's the question I got most, and that's the question that bothers creativity
researchers a lot, is how you judge your results, how you measure creativity. And that's also
why people never ask how creative this idea is. They try to make it into multiple dimensions and
make it easier for them to judge it. And that's also why we used both designers and the crowd.
We think this is a very good task for the crowd to judge it, because it is something consumer
product. People see a lot of chairs, chairs for children. If 100 people agree on that it's novel,
then it probably means it is novel.
>>: You said creativity is judged by how novel the idea is. How does it differ from the
originality?
>> Lixiu Yu: So, basically, being novel is similar to being original. People, it has to be both
original or novel and practical. Actually, there are other dimensions people consider sometimes.
>>: Why are you [indiscernible] sounds like two orthogonal dimensions that you are evaluating
this particular design.
>> Lixiu Yu: Yes, it has to -- an idea could be very novel or original, but it could be ridiculously
novel. It is impossible for us to manufacture or it's dangerous, so you have to consider the
second dimension. Actually, in the later study, we actually add more dimensions. It just
depends on what exam problems we used, we have to consider the context and who are rating
them and what's the best way to get an overall score to judge the quality. Okay, so this figure
just shows what people did. This is the design clocks, so that's the two original ideas, and the
third person trying to incorporate the features from two ideas and design a new clock. So to sum
up, in this process, we are trying to incorporate the force of what's happening in an individual
[indiscernible] and what's happening from a collective perspective, like as a population, how we
build culture, and we designed a distributive process which can promote distributed cognition.
And the aspect that I am personally interested in is, so people don't actually talk to each other in
person. You have 1,000 people. Once you use images and text of the ideas as a representation
for them to build on each other's work, it can really scale up. But then there's problems. So here,
you have the first generation seed ideas. Then you moved it to the following generation. It's
basically it's a converging process. It's trying to integrate it towards making it better and better,
but then actually before that stage, you have to get very good seed ideas. If the seed ideas are on
a very low basis, it basically will constrain the following ideas. And then also in design theories,
before you converge to something, you have to explore it. You're trying to find what may be the
other ways to go, what are the best directions to design your ideas? So the next projects, we
focused on the exploration phase, so in the context like you just want to generate ideas, how can
I help you to do that? So this time, we focused on exploration, reusing existing designs. So
using the previous designs --again, ideas seldom happened from scratch. They need to build on
the existing designs. That's also why distributed conditions are so interesting, because we have
accumulated all these designs, ideas and knowledge in the past, and how can we put people -like figure out a way to move forward to the future. So there are a lot of examples, but using
examples in the design process is actually not a novel idea. It happens a lot when people are
trying to generate new ideas, solve a problem, they would go out there, find some inspirations.
But using examples, even though it gives you some inspirations, it has a lot of -- it produces a lot
of negative effects. One of the typical ones is called conformity, meaning your new ideas most
times are very similar to other ideas you see. You make some variations, but somehow it
incorporates the previous features. It kind of reduces the existing ideas -- reduced their diversity.
So the other usual example is, especially in now the sharing culture, everybody can share their
ideas, and there are so many online idea repositories, then if we can figure out a way to use
examples, that would be great. But then, all these examples, which one do you choose? I cannot
give you 1,000 ideas and say, okay, generate new ideas, so that poses huge problems with how
can we select it and how can we process this information? Again, we can go back to the
creativity literature in psychology. Actually, they say there is a pattern between ideas, meaning
even though we have all these relevant ideas, actually, a lot of them are similar to each other,
meaning it's not like -- even though you're in a different field, you don't look similar to each
other. They actually share the deep structure, which is called analogy. So I'll give you an
example. This is the typical example using the psychology, is that a general tried to conquer this
castle, but then there are some roads, there are some mines under the bump. If you have a lot of
people walk through, it's going to trigger it, then the people will die. How can you conquer this
castle? So the strategy here is the general divided the army into multiple forces, and they
converge into the castle from multiple directions, then conquer this castle. A different problem
in the medical domain is that you have this tumor somewhere, and this laser can cure the tumor,
but it's too strong. If you just -- how to say it, if you just kill it with a big strong force of laser,
it's going to kill the skin surrounding it, too. So then strategy is you can divide the laser into
different small amounts and approach the tumor from multiple directions. So these two
problems are called analogies. So then you would think, how about we just present people
analogical examples, right? So, again, it has the problems of conformity. People would -- their
ideas would affect their examples, the specific features in some way. The other issue is you
would be surprised how difficult for people to recognize an analogy, even you tell them, you
show them these two problems, 50% of people cannot recognize the deep structure similarity.
So I'll use an example to show you what I mean by saying surface feature. So this is my nephew,
and he just turned one year old. He's super-cute, he's so smart, and he can make noise. He
cannot wait to know all about his world, and he spends some time with my mom, and so they
really like each other. My mom tried very hard to let him to see the world, the world of grandpa,
grandma, so after a month, he recognized this person actually called grandma, and then he started
to recognize some other things, so what we would ask him, what it is? And he said a bear. So
we showed him another, and he said, grandma! So why is it grandma, not a bear. You notice,
both of them are wearing a necklace. Both of them are happy. So as a baby that started to pick
up language, it cannot recognize the deep similarity between bears but the surface features
between these two completely different, I guess, objects. So this example shows you even
[indiscernible] intelligent, but we have huge problems with recognizing the deep structure. We
focus on the surface features. So but then a researcher showed that once people recognize the
analogy, they actually have -- they form some kind of a structure of thinking they call a schema.
It's called schematic thinking. They know that you need the structure. It doesn't matter what the
objects are, but if you share the structure that you are trying to convert and concur. You divide
something, convert it to the center and concurrent of the target. So the question we are trying to
solve in this project is how to apply the schematic thinking to the crowd, and how can we apply
it to the example, to design a distributed process? So we have this product, which is from
Quirky, that crowd design website. So it's a rake. The design there is a special slider on top of
it. After you collect these leaves, leaves sometimes stack on the rake, and you push it down, you
can get the leaves off the rake automatically. This is another product also from Quirky, invented
by two different inventors. So when you do barbecue, you can have the sliders preinstalled in
the skewer, and after it is done, you can push it and get rid of all of these things on your skewer.
So these ideas are very different. They have all these different features, they're in different
domains, but they actually share a deep structure. So then this project we are trying to solve is
we have all these examples. Some of them share these deep structures and how can we help the
crowd to generate better ideas from them? So the process we designed is the first step we need
to create a schema. The schema describes the deep structure. So we show these ideas, so we ask
people what common schema do these ideas share? So for these two, they are trying to attach
two objects by using a mobile slider. So this is how we operationalize the schema. And then we
asked people, just close your eyes, try to think, in what other situations might you need to detach
two objects from each other? So this set of designers or novice designers, crowd workers,
whatever we are focusing on, they don't see the original ideas. We give them these schema, so
we ask them to think in what other situations they might have such problems, and then the next
step, can you solve this problem by using a mobile slider? And this is another idea from Quirky.
It's when you chop things, and all these small things stick to your knife, the blade of your knife,
then you can design a clip to attach to your knife and get rid of it. So these are completely new
ideas, but it shares the structure of the previous two ideas. So this process I use to illustrate how
we did -- how the process should look like. Then we need to test this. First, we need to test the
idea that giving people schemas would provoke better ideas than giving them examples. So,
again, we get Turkers. We show them either examples or schemas and ask them to generate a
new idea. So, again, we have these problems of how to measure it. So this time, we don't give
them -- we didn't give them design problems, so they generate completely new ideas. They need
to figure out their own problems, in what domain, and solve it, what the ideas should look like.
So we add a second generation akin to the usefulness. So sometimes ideas are novel and
practical, but it's trying to solve a very achievable problem. It's not a good ideas. So we added
this third dimension to judge the overall quality of the idea. So I'm going to walk you through
how we ran the experiment. So, first, we designed a control condition to ask people to generate
new ideas. They saw three ideas, but we didn't ask them explicitly to use it, because they were
trying to recreate a real-life situation when people generate ideas, they just get inspirations.
They don't necessarily use them like in a very explicit way. So but then we also want to control
that the timing, the constraints we put on the experiments -- this is all the data, the experiment
design -- so we actually asked people to generate new ideas by using examples. And the last two
conditions, we focused on schemas. We asked them to generate an idea by using design rules, so
this time, we didn't use schemas, because we didn't want to introduce new terms to the Turkers
that they didn't understand, so we used design rules. So they actually have this schema and
they're trying to generate new ideas by seeing both the example and the structure. So in the last
one, we only showed them this design rule. So we are interested in what kind of manipulation
might help people to generate better ideas. So in the first two example-based conditions, on
average, the quality of the idea is 2.5, and in the schema-involved conditions, you can see that
we improved the quality of our ideas by about one point, I guess. And so from the regression
analysis, we found that showing people schemas can significantly improve the quality of their
ideas. But then, the next question is how can we generate schemas. In that condition -- sorry, in
the previous experiment, we have designed experimenters. We designed schemas. We also want
to make everything -- can be conducted by the crowd. So we're trying to find a way for people to
generate schemas, and there were a lot of studies in the past of people trying to make them to
summarize the problems or write down the underlying principle or generate direct visual
representation of the problems, and all of them have failed. It is very hard for them to recognize
deep structure of ideas. But there is some -- in the past literature, there is some projects worked.
One is, you show people multiple examples that are analogies to each other. They share the
same deep structure. Then, it can help people to realize the commonality in the structure, and the
other one that worked is showing people constructive examples, which is different, which doesn't
share the structural similarity.
>>: The sets are already known?
>> Lixiu Yu: Yes, yes. I'm going to get to that question later. Yes, so the sets are already
known. We picked -- we want to say what situations people are more likely to generate the
schema, so we controlled the examples we selected. They have to be analogous to each other.
So this idea, so it's used to keep your ear buds stable when you walk or run. The second idea -these are all real ideas from Quirky -- is the product about when -- the dishwasher, the glasses
sometimes get broken. They designed tethers to attach the glasses to the dishwasher so that it
can be stable. The third one is your head will move around when you sleep on a plane or a bus is
like a strap thing to attach your head to the back of the seat. So it's very cute products. They're
very popular on Quirky. They actually make a lot of money from these products. So they are
different products, right, but they share this structure we call the schema, so we also add different
products. This one is like you make the power strip which can change shapes, so it's completely
different from the first three. So we either show people one of them or two of them or three of
them, the first two or first three, or we replace one of them with a last one to add a contrasting
example, to see how can people generate the better schemas. And this time, we found as you
increase the number of analogous examples, people are more likely to recognize the deep
structure similarity. They can generate very good schemas. But when you add contrast
examples, you kind of distracted them from recognizing the schema.
>>: [Indiscernible].
>> Lixiu Yu: No. We didn't tell them. We just showed them the set. Okay, then now we want
to close the loop. Remember, my central research goal is to assign tasks to multiple people, to
design distributed structure, so we're trying to close the loop to take these schemas generated by
the crowd and to use as input to ask another group of people to generate a new idea, to see
whether the effect we got in the previous experiment is because that's something we produced.
We want to make everything natural to be conducted by the crowd. So we selected good and bad
schemas from the previous experiment, and we asked people to generate the ideas by either,
again, observing the examples or using bad schema and examples or using good schema and
examples. We want to see whether the quality of schema matters and whether people can
generate better ideas by using their colleagues' output. So again, this time we found that there
was no significant difference observed between the examples of bad schema and examples, but
when you showed people good schemas, produced better coworkers and examples, they can
produce much better ideas. So these three experiments together composed a distributed process
that suggests that we can break this process, starting from examples -- a set of people can
identify analogous examples, a different set of people can generate schemas for these examples
and a third group of people can generate new domains from these examples, and the last group
can generate new ideas in these new domains. We didn't really test every step, but we actually
ran a three-step experiment, but it suggests that we can break it into multiple steps to elicit as
many ideas as possible and promote the quality. Then, now I'm getting back to your first
questions. We -- as experimenters, we selected the set of the examples. Then you're constantly
having -- like how can people even identify such examples? So that's the idea we're working on
currently. So we used the crowd to -- the first approach we tried is how can we get the crowd out
there to find our analogous examples. So we showed them an idea, showed them a schema, told
them go up there, find an idea, share this structure. So they go to Quirky, they have these ideas,
and they try to find analogous examples. The accuracy was much, much higher than they just go
out there and find examples, analogous examples. But the question, again, is even though we
have this [indiscernible], it still takes a long time for them to go through a database with an
unlimited number of ideas. So we're trying to combine the natural language processing
techniques with the crowd, how to efficiently identify the existing set of analogous examples as
ongoing work. Okay, so to sum up, in this project, we are trying to figure out the new
representation of creative content, and among the existing examples to make it as a better source
for people to generate new ideas. Also, it suggests a distributed design process which we can go
from concrete content to abstract and then go down to the concrete ideas again. So future
research, so all my research has been based on the idea of a distributed combination, and so in
distributed cognition, people actually think about the mind in two ways. One is you can look for
the mind-like property from a large group, like how can you treat person as a neuron in your
mind, and do you make them actually work in some way to do something great? But the other -like the metaphor can be run in another way around, meaning we can treat the mind as a society
of persons, like each node can function as a person, kind of like handlers of machines and
connected in specific ways. And in that way, we can borrow -- or it's my interpretation, is that if
you think about it from these two ways, then whatever we learned from minds in society, the
groups, the larger group of people, and apply it back to the mind, to make individual -- to be
more creative, to solve problems better. So along the first lines, there are a lot of directions I
would like to pursue. One is continue the current work and search for creative content, as I said,
like how can we use the crowd, the humans' cognition and the machine learning to find the
creative ideas. So, currently, if you're trying to design the last ideas, I showed you how to detach
the garlics off the blade of your knife. Currently, if you search whatever related to that idea,
feature, like a knife, cooking, garlic, everything I tried, you would have all these ideas. They're
not really helpful for you to solve that problem. But if you search with that structure, like
detached two objects by using slide objects, these are the results I find. I did a test. The middle
one is very interesting. It's like a guy trying to ask a girl to pay for everything he purchased.
Actually, that's a metaphoric way to detach something. You annoy a girl, but it's not really
helpful. So then the goal of this project is how can we actually put something, search in some
way and get these ideas as a return. So the second idea, I think I need to go back -- sorry. So
now, still, we need motivation. We need all these crowd inventors, even though in Quirky's
incentive, they got at least 340,000 inventors working there. And how can we make designs
automatically? When people are trying common products, or especially for the unsatisfied
consumers, they would say what's wrong with this product? Then how can we integrate this
creative content from customer reviews and automatically reinvent the product? I think that is
another direction I would be interested in working on. So, secondly, remember we can think of
the mind in terms of a lot of different machines there that you can treat it as a society that would
then mean by purchasing future research. So that's the assumption. We can think individual
cognition that way, but it is good if we actually find some things, because once you make it a
study in a more structured way, you can get internal agencies -- internal neurals, internal
organizations, into coordination with external structure. That's why I think it might be
interesting to study it for that direction. So I'm using examples to illustrate what I mean when I
say internal, society of our mind. So this is the work I showed you. I used each design was
generated by a crowd member, and they designed these ideas in parallel, and later it integrated
into fewer and better ideas. This work was done around 2010, and in the exact same time,
Stephen Dunn from Carnegie Mellon University, he was in Stanford back then. He did this
work. It's on the topic of parallel prototyping. So, basically, the founding of that work is that if
you ask people to generate multiple ideas at the same time and then later trying to narrow that
down to a particular prototype, you find that people take the features, the multiple features, from
the previous multiple variants and generate a final design. They compared it to an iterative
process, where people just have one single idea and they keep working on it to get feedback and
generate final designs. And the parallel prototyping led to much better ideas than the iterative
process. So at this point, you might notice the remarkable similarity between these two works,
right? We did it separately, we never coordinated. He's trying to study a single mind. We are
trying to study a distributed mind, but we follow the same thought, like how it works, the parallel
thinking, the divergent thinking. I guess my point by using the example is if we don't know this
work, we only have my previous work, we can actually try to use that perspective to understand
and figure something like this. And if I haven't done that, and after reading this work we can
borrow whatever finding there to apply to the crowd. So that's my future research, is we can use
this kind of thinking to study individuals' minds, and all of this, the random thinking, exciting
future research, I'm not going to go through each of them. Future research is always exciting.
And I'm listing a few of them. One is the analogical thinking to be [indiscernible] the crowd,
like give them a structure, ask them to generate new ideas. Then, how can we apply it to the
single mind. People, most of the time, they get stuck to the ideas they have. They cannot think
in another way. How can we put them in different contexts and mimic them as a multiple person
working in the same time to think of multiple analogies to be able to view the same ideas in a lot
of ways. The other idea is, so people have been trying to figure out like in Wikipedia or in other
kind of communities, peer products and communities, how can we get people to contribute to
certain content? Like in Wikipedia, a lot of popular topics, 1,000 people are trying to improve it,
but there are some other areas, nobody ever works on it. But the same thing to individuals,
right? Sometimes, we have tasks, we just don't want to work on it. It takes forever for us to get
to us, or when we're trying to solve problems, we're just stuck into a particular direction. We
cannot think of any other ways. So in the peer production community, there's work done at
Minnesota University. They actually designed these things called a suggest bot. They're trying
to reduce the contribution cost like to suggest to you this area is already full, you should work on
this area. So it detects, like uses some algorithms to detect the areas that need to be worked on
and suggests like what you should work on. So maybe when we think about individuals, we can
help them to find the thought land, spot. So by doing that, when we figure out what's going on in
the mind, what's going on out there, so we can get them both in better coordination, so we can
have the real distributed mind can be contributed by individuals and can be built on by others
anytime, anywhere, in any way we want. Thanks.
>>: So it's a very interesting topic. Thank you. So one of the questions I have about some of
your last few slides is that so you worked closely with the Design School, and in the discipline of
design, one of the things that people are taught is that so before they go to a crit, a design crit,
they have to come up with three independent ways to present the same idea. It's a very standard
procedure. They might get a larger frame to split it amongst three teams, but a design student
will be trained to be able to come up with three independent ideas. And I wonder -- because that
gets a little bit to some of these questions you were asking, is how do we get people to be better
at that kind of independent ideation of their own? Do you know if anybody has kind of looked at
-- because they go through some training for this, but in the end, are designers then better at
doing -- do they get less stuck in their own ideas than people who do not have that training? Do
you know if anyone's looked at that?
>> Lixiu Yu: Well, that sounds exactly like what Stephen Dunn has done, right? The parallel
prototyping. So that work, they specifically focus on advertisement design. It's not designers,
real design students, versus novices, how can you make them to come with multiple ideas and
then later on convert to single ideas. It is related work. Yes. Andrew?
>>: I had something -- so [indiscernible] focus on design, which is a really appropriate way of
using some of these techniques, and so I was wondering if you could speculate on how this might
apply to things that are not as practical as design things. Like, objects are useful, but more things
that are artistic, for example, or that certain kind of aesthetic, or even jokes. Could you do these
kind of processes for producing funny jokes or beautiful things and so on? I'm just curious, like
if you think about this, how might you apply it, or whether you think some of these specifically
for things that are practical, or do you think it extends beyond that?
>> Lixiu Yu: So there is a continuum, right? If there's two dimensions, one is original novelty,
like a lot of creative stuff plays a big role, like painting, artistic stuff. The other direction is
practical. You can go either way, right? The art thing will be in this extreme and copyediting
will be on this extreme. And so we have been working on the consumer products and also
[indiscernible] working complex problems, like social problems, like in open [indiscernible],
people are trying to get the crowd to solve social problems, like how can we get people to be
more healthy, how can we get people to engage in community more? So in that community, they
actually tried analogies, and it worked very well. So you will notice, when people generate new
strategies, they're actually trying to borrow metaphors from the other domains. So these are the
two field works. I'm not sure it's going to work. I personally think, as the random factors, the
creativity increases, it will work less and less effectively, because there's higher uncertainty. In
particular, painting, we don't even know how to judge it. I don't even understand Mona Lisa's
smile, that particular painting is good. I still don't understand it. I went to the Paris museum,
people say it's good, and maybe it's good. So I don't think it's going to work in that area, but I do
think in between -- it depends on how we design the task. There are lots of studies in artificial
intelligence and in psychology, how can we convert the uncertainty into certainty, how can we
treat the random thinking, creative thinking by conducting in a more structured way? So I do
think it will work in that way, so even in Quirky, some products are very practical, like a
[indiscernible] product, but some other products, they're kind of creative. When you think about
it, it is very interesting, so it doesn't matter who we involve in this process, how we design the
task, we have to figure out what will really need to happen if we design something. Yes, great.
>>: I have a question related to the fact that designers go to school and they learn their practice,
and they get better over time. I'm thinking about the kind of improvement that happens in an
individual as they go from not being able to design very well to being able to design very well,
and even better as they get more and more experience. What do you think would be the process - or what do you think would happen in the crowdsourcing example, where you have these
Mechanical Turkers that are designing things today, and now you keep giving them designs.
What's going to happen in a few years or a decade from now? Will they be doing better designs
in a decade? Or because there's this churn, people maybe aren't getting feedback, will they be
doing the same quality of designs in 10 years that they're doing today? Even at the baseline -- at
the baseline, they seem to be starting off better, what will happen in 10 years?
>> Lixiu Yu: Well, we are talking about two communities. One community is like Quirky,
incentive. They're a stable workforce there. The inventors, even though they are novices, but
they constantly participate in the design process, and the website, their organization is actually
trying to help people to build these different ways of thinking. I do think from the organization
perspective, if they can figure out something to help people to develop the skills which can scale
up, like design students learn from design school, they can become better over time, but the
community I guess I have been focusing on, I think it's most interesting is how can we use these
transient workers, one-time workers? They don't really care about making designs or achieving
complex work as a lifelong profession. They are just interested at that moment, I want to
participate in this kind of job. That matters less about how can we educate our trained
individuals and more about how can we design systems, like whenever coming in, we can plug
them temporarily in to conduct this process. That's the job of our researchers, to design this
system that can incorporate transient workers. Any other questions? Thank you very much.