User Feedback in Design for Emerging Markets: Methods and Influencing Factors
ARCHNES
by
MASSACHUFTTf IN!TT JTE
1
OF TECHNOLOLGY
Jasmine Florentine
JUL 3 0 2015
Submitted to the
Department of Mechanical Engineering
in Partial Fulfillment of the Requirements for the Degree of
LIBRARIES
Master of Science in Mechanical Engineering
at the
Massachusetts Institute of Technology
June 2015
C 2015 Massachusetts Institute of Technology. All rights reserved.
Signature redacted
/
Signature of Author:
Signature redacted
,
Certified by:
Dtp/rtment of Mechanical Engineering
May 18, 2015
4$aria Yang
M
Associate Professor of Mechanical Engineering
T)1esis Supervisor
Accepted by:
Signature redacted
David Hardt
Professor of Mechanical Engineering
Chairman, Department Committee on Graduate Students
User Feedback in Design for Emerging Markets: Methods and Influencing Factors
by
Jasmine Florentine
Submitted to the Department of Mechanical Engineering
on May 18, 2015 in Partial Fulfillment of the
Requirements for the Degree of
Master of Science in Mechanical Engineering
ABSTRACT
Designing products for emerging markets in the developing world can be difficult. Design
theory holds that understanding the user better can improve the success of the product, however,
formal user research approaches designed for conventional markets may not be effective in
emerging market scenarios. This thesis explores three factors that can affect the quality of user
feedback: the method used, the demographic of the user, and the type of prototype presented to
the user for feedback. Data is collected from two different case studies conducted via field
studies in India. The first case study investigates a novel solar technology, and the second
centers around a technology to improve rural agriculture. In these case studies, results show that
the user research method used yielded the type of feedback expected in conventional settings,
although users had difficulty with more abstract concepts. Demographics played an important
role in that younger and/or more educated users were more open to giving criticism and asking
questions. Users were generally able to understand that prototypes were only a representation.
3-D prototypes sometimes caused users to fixate on certain form factors, whereas 2-D prototypes
allowed them to interpret concepts more ambiguously.
Thesis Supervisor: Maria Yang
Tile: Associate Professor of Mechanical Engineering
3
4
Acknowledgements
I was supported through many people throughout this endeavor, to each of whom I owe my
greatest thanks. I would like to first thank the MIT Tata Center for making this project possible,
and providing me with the resources, support, and education necessary for such an endeavor.
My extreme gratitude goes out to my amazing advisor Maria Yang for guiding me through the
process of research, being extraordinarily patient with my many questions, and being an
incredible advisor. Thank you also to the students and post-docs of Ideation - Qifang, Anders,
Bo, Geoff, Carmen, Susan, Ned, Jim, Mike, Maxime for making graduate school awesome, and
especially to Jesse Austin-Breneman who was always available to bounce ideas off and edit my
work. I would also like to thank the amazing UROP's who contributed to this project: Catherine
Fox, Kendall Helbert, and my trekking buddy, Ben Collins (without whom, field studies in
Ladakh would have been impossible).
My thanks also go out to the Solar Thermal Fuel team (John McGann, Alison Greenlee, and
Priyank Kumar, and Prof. Jeffrey Grossman) and the Soil Diagnostics team (Ron Rosenberg,
Soumya Braganza, Mark Jeunette, Ankanksha Nagpal, Chintan Vaishnav, and Prof. A. John
Hart) for making our studies in India not only productive, but incredibly fun. Thank you also to
Stephanie Houde and Subarna Basnet for all of their fantastic advice, and Prof. David Gordon
Wilson for sparking my interest in solar technology for emerging markets.
This thesis would have been impossible to complete without the many kind people who helped
me in India. I interviewed and observed many people, who were all kind enough to let me into
their homes and lives. In particular, I would like to thank Susmita Misra and Jagannath for their
assistance in Delhi, Mahendra Kumar and Pooja Mukul for our work in Jaipur, and Tunduk for
helping us out in Ladakh. I would also like to thank the Deshpande Center for everything they
did to help out the Soil Diagnostic team in Hubli, with a particular thank you to the incredible
Manjunatha.
I would like to give a very special thank you to Fatima and her amazing family (especially Fariha
and Asma!) who I stayed with during my month in Leh. They went over and beyond the call of
duty and made me feel not only comfortable, but like part of their family. They invited me to eat
dinner with them, took care of me when I was sick, watched Bollywood movies with me, and
even helped me with my field studies.
Lastly, thanks to my family for supporting me throughout; particularly my mother who had to
suffer through long phone calls, and my aba who worried nonstop about me whenever I was
doing field studies. Thanks to my wonderful siblings, Evelyn, Michelle, and Ethan for not
driving me crazy. And thanks to Nate Zuk for his infinite patience and delicious scones.
5
6
Table of Contents
S Introduction.............................................................................................................................
M otivation ........................................................................................................................
Research Questions ........................................................................................................
Contributions....................................................................................................................
11
11
11
12
Related W ork ..........................................................................................................................
2.1 Introduction ......................................................................................................................
2.2 Em erging m arket users.................................................................................................
2.3 D esign for em erging markets ........................................................................................
2.4 U ser research m ethods ...................................................................................................
2.4.1 Conventional M ethods...........................................................................................
2.5 U se of Prototypes in user research ...............................................................................
2.5.1 U ser feedback on prototypes .................................................................................
2.6 Case study methodologies.............................................................................................
2.7 W hat is the research gap ................................................... .......... ................. .............. .
14
14
14
15
17
17
19
20
20
20
1. 1
1.2
1.3
2
.
3
Case Studies ............................................................................................................................
3.1 M ethods............................................................................................................................
. . .. . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . .
3. 1.1 WhyCaseStudies 9
3.1.2 Case Study M ethods ...............................................................................................
3.1.3 Defining prototypes...............................................................................................
3.2 Case Study 1.....................................................................................................................
3.2.1 Background................................................................................................................
3.2.2 Problem Identification...............................................................................................
3.2.3 Challenges of eliciting user feedback....................................................................
3.2.4 Categorizing the Types of Prototypes used ..........................................................
3.2.5 Case Study 1 Discussion ........................................................................................
3.3 Case Study 2.....................................................................................................................
3.3.1 Introduction ...............................................................................................................
3.3.2 Process.......................................................................................................................
3.3.3 Case Study 2 Discussion ........................................................................................
3.4 Limitations .......................................................................................................................
4
Results .....................................................................................................................................
54
4.1 Classifying Types of Feedback ......................................................................................
54
4.2 Coding the Feedback....................................................................................................
56
4.3 V ariables A ffecting Feedback......................................................................................
57
4.3.1 Feedback Results...................................................................................................
58
4.3.2 Lim itations.................................................................................................................
61
4.4 Qualitative Effect of Prototypes....................................................................................
61
4.4.1 People were able to extrapolate from prototype to images. ..................
62
4.4.2 Storyboards helped users visualize the product more holistically in some cases, but
caused confusion in others.................................................................................................
63
4.4.3 U sing the right prototype........................................................................................
63
7
22
22
22
22
22
24
24
25
26
33
34
40
40
40
50
53
4.4.4 D id the narrative aspect m atter?......................................... . .............. ...................
4.4.5 Confidence in feedback ...........................................................................................
4.5 Other A spects Affecting Feedback ...............................................................................
4.5.1 Fam iliarity w ith Solar Technology.........................................................................
4.5.2 M icro-entrepreneurs ...............................................................................................
4.5.3 G ender .......................................................................................................................
4.5.4 Cultural Background of the Interview er................................................................
4.5.5 Translation.................................................................................................................
. .
63
64
64
64
64
65
65
65
5
67
General D iscussion .................................................................................................................
67
5.1 Observed differences....................................................................................................
68
5.2 Other Takeaw ays..............................................................................................................
68
5.2.1 Intro ...........................................................................................................................
68
5.2.2 G etting realistic responses......................................................................................
69
5.2.3 D ifficulty w ith abstract ideas..................................................................................
5.2.4 Making sure to interview a variety of people - users and other stakeholders; get a
69
very different story................................................................................................................
70
5.2.5 Observations combine w ith interview s..................................................................
70
5.2.6 G iving m ultiple choices to choose from ................................................................
70
5.2.7 V illage Politics ......................................................................................................
70
5.2.8 Looking out for misunderstandings.......................................................................
71
5.2.9 Attitude ......................................................................................................................
71
5.2.10 Translation ...............................................................................................................
71
5.2.11 Good local contacts................................................................................................
71
5.2.12 Y ou are a tourist attraction ....................................................................................
72
5.2.13 A daptation................................................................................................................
72
on
the
ground..............................................................................................
Being
5.2.14
6
Summ ary .................................................................................................................................
6.1 Conclusions ......................................................................................................................
6.2 A reas that need to be further researched ......................................................................
6.2.1 Controlled experim ents w ith prototype types.........................................................
6.2.2 Moving towards participatory design with more complex technology ..................
6.3 Ram ifications ...................................................................................................................
7
Bibliography ...........................................................................................................................
8
73
73
74
74
74
75
76
List of Figures
Figure 3-2: User in Ladakh demonstrating a solar cooker........................................................
28
Figure 3-3: Storyboard used to demonstrate solar thermal fuel as a portable device................
29
Figure 3-4: The "magic box" physical prototype .....................................................................
30
Figure 3-5: Interview group, onlookers, and translator with the "magic box"..........................
32
Figure 3-6: Examples of non-narrative images. Left: ways the STF can be used for cooking.
Right: Potential form factors STF-based heating......................................................................
33
Figure 3-7: Classification of the Solar Collection Storyboards, the STF Applications, and the
34
"M ag ic B ox"...................................................................................................................................
Figure 3-8: Conducting group interviews in a farming village near Hubli................................
41
Figure 3-9: Product attribute cards depicting (from left to right) durability, time taken, and labor
44
in ten siv en ess .................................................................................................................................
Figure 3-10: Image from the first workshop.............................................................................
45
Figure 3-11: Storyboard used to demonstrate collecting soil for testing as it is currently
p racticed ........................................................................................................................................
47
Figure 3-12: The moderator and translators roleplaying with the physical props ..................... 48
Figure 3-13: Classification of the Soil Collection Storyboards and the Soil Collection
Mo ck U ps......................................................................................................................................
9
49
List of Tables
Table 3-1: Comparison of the locations visited ........................................................................
27
Table 3-2: Comparison of different user research methods......................................................
36
Table 4-1: Categorizing the different types of feedback ..........................................................
56
Table 4-2: Categorizing the different studies.............................................................................
58
Table 4-3: Analysis of feedback versus method........................................................................
59
Table 4-4: Analysis of feedback versus prototype....................................................................
60
Table 4-5: Analysis of feedback versus demographic ...............................................................
61
10
1
1.1
Introduction
Motivation
Designing products for markets in the developing world can be difficult and success can be
elusive. Many well-intentioned high profile products like the Life Straw, Play Pump, and One
Laptop Per Child were not as successful as had been hoped for [1] [2]. One often cited reason
that products for the developing world do not gain as much market adoption as desired is a
misunderstanding of the user from the very beginning of the project [3]. User centered design is
one way to try to avoid such failure - it is a successful, often used approach whereby user needs
and wants drive product development [4]. To understand user needs, quality feedback is critical,
which is why many structured methods for eliciting user feedback exist. Although we live in an
increasingly connected world, some have argued that emerging markets differ enough from
developed world markets that conventional design methods no longer necessarily apply [5] [3].
For a designer working from the US, there are not only barriers like language and accessibility to
users, but cultural differences that influence how users may give criticism, or lifestyle
differences which might affect a designer's understanding of even day-to-day activities. Even
for native designers, differences of lifestyle, education and socioeconomic class can pose
significant barriers. For instance, India is such a large country that a university student from
urban Delhi may have little in common with an older rural farmer in Ladakh - they may not even
speak the same language!
This thesis explores the factors influencing user feedback in emerging markets, with the
goal of helping designers gain better quality feedback. Literature has shown that the type of
method used can affect the type and quality of feedback elicited [6], but there is little information
describing using these methods in emerging markets. We look at two case studies of design
projects in India to gain a better understanding of how designers can elicit quality feedback in
emerging market situations.
1.2
Research Questions
There are many different aspects of gathering quality user feedback, but for the purposes of this
thesis, we focus on two main questions:
11
1) How does the method used affect the quality and type of qualitative data gathered? For
our studies, we focus on ethnographic methods such as observation and interview, as well
as focus groups and workshops.
2) How does the demographic of the user (age, education, social class, area of India,
urban/rural, gender, etc.) affect feedback?
3) How does the type of prototype used affect feedback?
The "type" of prototype can be defined many ways - by fidelity, role, implementation,
etc. [7][8]. For our studies, we were interested in particular by the dimensionality of the
prototype (whether it was a 2D dimensional prototype such as a storyboard or 3D such as
a physical prop), and the level of narrative conveying the concept.
1.3
Contributions
Existing design methods need to be modified to fit the unique situations of emerging markets.
This thesis focuses specifically on the situations in which student designers visit another country
for a brief period of time, and must do the user analysis aspects of their work during these visits.
Our work focuses on the early stages of the design process, when designers are still working on
understanding their potential users, and evaluating early stage concepts. From the two case
studies, we draw some conclusions about how the method and prototype used can affect the
feedback elicited from users.
Our hope is that this work can inform future projects. Quality feedback is important to
understanding users, and in designing for emerging markets, can make or break the product [3].
By improving the early stage of the design process itself, we hope to improve the ultimate
outcome of a design project, by giving designers the tools to design appropriately for their target
users.
12
13
2
2.1
Related Work
Introduction
The two main bodies of existing literature of interest are in design for emerging markets and on
design methodology, specifically with a focus user analysis methods and use of prototypes.
2.2
Emerging market users
One question that needed to be addressed early on was how do users in emerging markets,
specifically in India, differ from those in the United States? There is no simple answer to this
question, since emerging markets encompass a large number of countries, cultures, and
socioeconomic groups. Even in India alone, a designer may find him or herself working with an
illiterate rural farmer or a middle class educated urban housewife. Because of these vast
differences, it is difficult to come up with a single unified set of differences between these users
and American users. Urban middle class users in India may be much more similar to their
American counterparts than they are to their compatriots at the bottom of the pyramid.
Factors such as culture, language, purchasing power, and infrastructure all add new
barriers to the adoption of products. Not only that, but to treat emerging markets as one
monolithic entity with homogenous needs would also be an error. Just as markets in the
developed world differ, so do those in the developing world. Fortunately, there is already some
literature about the unique circumstances that are at play in using the conventional methods of
user analysis.
Maunder, Marsden, Gruijters et al. discuss some of the difficulties with applying current
methods to the developing world [9]. Since some users in the developing world are not exposed
to high tech products and solutions, they may find it difficult to understand or imagine a
technological concept presented by a designer, or even understand the value of the design
process itself. Given that it can already be difficult asking users to speculate what they want in a
product can be difficult even under normal circumstances, asking someone with very limited
exposure to technology adds another barrier.
Honesty, already an issue in conventional markets, also becomes more complicated in
emerging markets. Tao, in developing a
cost
-O autoclave for rural health clinics in Nepal
discusses some of the factors unique to his experience in the developing world setting that made
14
it especially difficult elicit honest feedback from users - such as the presence of superiors, the
presence of non-locals, and the donor culture [10].
Yeo argues that one cultural factor influencing usability testing is the power distance
[11]. This is something we were not able to control for in our studies, since our users may have
felt a power distance with us and/or our translators as a result of socioeconomic status,
education, or other factors.
2.3
Design for emerging markets
Scholarly literature on product design for the developing world is still somewhat limited;
however, there are a number of papers that approach some of the issues of the difficulties of
designing for these markets, and of the low rate of adoption of products. Donaldson focuses on
Kenya, and found that a formal design process was generally not used there [12]. She argues
that for product design to support economic growth in less industrialized countries, the design
process needs to be user centric, and match local conditions such as corruption and inadequate
infrastructure. Mattson and Wood review a number of papers on the topic of design for the
developing world, and arrive at nine principles [13]. The first two they outline are most relevant
to the early stage design process in encouraging co-design, and testing the product in the real
world during all steps of the process. The Demand-Driven Innovation handbook also outlines
several principles, albeit from a higher level, business oriented perspective [2]. Nieusma and
Riley make the case that many development projects place too much emphasis on the
technology, to the point of ignoring economic, social, and cultural considerations and can even
do more harm than good by increasing social injustice [14]. Maunder, Marsden, Gruijters et al.
examine how the broader context of the culture the designer is working within may impact the
success of the product, and how the designer must consider broader environmental and
socioeconomic impact [9].
Gregory Tao is a designer and entrepreneur, and in an interview he elaborated on some
of the difficulties he encountered in creating an autoclave for rural Nepal, beyond what he had
already documented in his thesis [15][10]. The group's critical assumption was that the root of
the autoclaving problem was an issue of educating people and making them easier to use.
However, what they gradually found out through in situ field research was that the real problem
was convincing people why they should use an autoclave. This highlights the importance of the
buyers and the end users themselves recognizing that there is an actual need. This, along with
15
difficulties along the supply chain and the reluctance of repairmen to travel to rural areas made
the adoption of the autoclave difficult.
Tao suggested several design methods for future projects based on his experiences. First,
he suggested co-designing in such a way to put the designer on a level playing field with the
users, both by bringing in the correct tools and by keeping the initial idea rough enough for the
users to contribute to. He emphasized the importance and the difficulty of getting honest
answers, because it is difficult for users to envision how they might use a future product, because
they may be reluctant to give negative feedback, and because they may feel uncomfortable.
Tao's suggestion was to engage with them in a way that they didn't feel they needed to give
falsely positive feedback, by disarming them and ensuring them that the designer is not
particularly invested in this idea. He suggested bringing a very rough model of what the product
might look like and having users play with it directly, with the goal of getting them to articulate
their needs and possible solutions. We took Tao's suggestions to heart in planning on field
studies and prototypes.
Vainio, Walsh, and Varsaluoma discuss some of the difficulties in cross-cultural design
in developing a mobile learning tool for South African schools, and conclude that there were
enough culturally dependent issues that localizing the design improved the user experience [16].
Although not necessarily specific to design for emerging markets, Vatrapu and PerezQuiflones found that the cultural background of the interviewer affected the type of user
feedback elicited [17]. When doing usability testing with Indian participants, the participants
spoke more freely, found more problems, and made more suggestions when the interviewer was
also Indian. However, a study by Oyugi, Dunckley and Smith evaluated conventional "Western"
design methods across three cultures (UK, Kenyan, and Indian), and found that "Western"
methods were less effective in other cultures [5]. In fact, they found that even when they had the
users and evaluators be of the same culture, the Kenyan and Indian groups still had poorer results
with the conventional methods.
MIT's D-Lab recently released a document outlining a framework for user research in the
developing world aimed at individuals and organizations working in the field of international
development [18]. A rough draft of the document was used to guide our user research process.
Their document provides a wealth of information on the practical aspects of planning for and
executing user research in the field, starting from outlining the research plan, to documenting and
16
processing it. Nielson offers a brief but practical guide to international usability testing (with a
focus on HCI), and emphasizes the importance of visiting the country one is designing for; he
also makes some practical suggestions with regards to overcoming language barriers [19].
Another approach to improving the adoption of products, suggested by Austin-Breneman
and Yang is to focus on micro-entrepreneurs [20]. They also suggest a list of design guidelines:
multi-functionality of the device, ability to generate income, educating the consumers about the
value proposition, and establishing a reliable brand identity.
Our work focuses less on the overall product life cycle process and more on the early
stage, but the literature helps us understand the full context of what makes product design and
adoption so difficult in emerging markets.
2.4
User research methods
2.4.1
Conventional Methods
A number of structured processes for understanding user requirements already exist for
developing products in the industrialized world, but they were not creating with the unique
circumstances of the developing world in mind. Before exploring how effectively these
conventional methods work, it is important to understand the existing methods. Below are brief
summaries of the methods we focused on in our studies, drawn largely from the work of Courage
and Baxter [6].
2.4.1.1 Interviews
Interviews are one of the most common methods used to gain user insight. They provide a large
amount of detailed data, but because of the time intensiveness, from a small sample size.
However, they can be used to gain an overall context and understanding before moving on to
other usability studies. Some of the things to be aware of are unintentionally creating bias
through the wording of the questions, and honesty of the feedback. Although people may not be
intentionally dishonest, they may try to please the interviewer, or present themselves in a better
light. Courage and Baxter also warn against asking users to brainstorm solutions or challenges,
since they may not understand the technical requirements of the concept in question.
17
2.4.1.2
Focus Groups
Focus groups are good for getting data from multiple people more quickly than one-on-one
interviews (although still not at the level of statistical significance one might get from a survey).
The group setting also encourages people to bounce ideas off of each other, and to feel
encouraged to say things they might not bring up in a one on one interview. However, they have
some drawbacks. Because of the larger number of people, it can be difficult to get more in depth
than in one-on-one interviews. Additional challenges are that users might influence one another
in their responses. A good moderator can help avoid one person controlling the entire group.
Courage and Baxter also warn against asking focus group participants to predict what features
they might want in a product - similar to what they warn about asking users to brainstorm in
interviews. Another important point is that in a focus group, there is not the opportunity to
actually observe users performing a task or in their environment. This could lead to
discrepancies between what users say and what they actually do.
2.4.1.3 Ethnographic observations
Observations are good for watching users first hand in their own environments - especially since
asking people to describe their behavior can be less accurate. They are good for building
context, understanding inconsistencies between what people say and do, and collecting rich, in
depth data [6][18]. However, they are time consuming. Some of the things to be aware of are
that users may act differently knowing they are being watched, although a longer period of
observation can mitigate this. Observers may introduce bias into the data as well in how they
understand and record what they observe. There are a number of methods, but the ones we
focused on in our studies were deep hanging-out and process analysis. Deep hanging-out does
not require user interaction, but rather simply observing the user. Unlike pure observation, there
is some focus to what the observer is focusing on. Process analysis involves observing the users
conducting a process, and having the users walk the observer through the process. In all of these
cases, flexibility is required, since these methods can really only guide the process so much.
2.4.1.4
Workshops
The term "workshop" can encompass a variety of meanings, but for our studies, it meant a doing
a set of interactive activities with a group of users. The studies we looked at that used
workshops differ enough from focus groups that it makes more sense to consider them
18
separately; therefore we use the term workshop largely to differentiate the method from
conventional categories with more prescribed methods. There is precedent for using these
customized methods in emerging markets, such as the participatory design workshops an NGO
ran in Sri Lanka, or Apple Computer Inc.'s work with PictureCARD in India [14] [21].
2.5
Use of Prototypes in user research
There is already a great deal of research into different types of prototypes in the area of Human
Computer Interactions (HCI). Much of the research in this area also applies to prototyping in
product design. Houde and Hill broadly define prototype as "any representation of a design idea,
regardless of medium" and develop a model to help describe aspects of a certain prototype along
three dimensions: role, look and feel, and implementation [8]. McCurdy and Connor take it
further and identify five dimensions to characterize thefidelity of a prototype: level of visual
refinement, breadth of functionality, depth of functionality, richness of interactivity, and richness
of data model [7]. These dimensions help to characterize the types of prototypes later in this
thesis.
There are many papers in HCI discussing the use of paper prototyping over actual
software. Sefelin, Tscheligi, and Giller show that using paper prototypes does not significantly
change the amount of or type of criticism [22]. Virzi, Sokolov, and Karis argue that paper
prototyping can affect the type of problem found [23]. However, we believe that the situation in
the developing world can be different due to lack of exposure to technology. That said, Parikh,
Ghosh, and Chavan worked with Indian villagers in developing a user interface for a financial
program [24]. They found that while villagers were initially confused by the paper prototypes,
they understood them after some explanation. However, since they were looking at a mockup of
a computer interface rather than a physical product, their conclusion cannot be generalized to all
product design.
Tohidi, Buxton, Baecker et al., found that presenting users with multiple design concepts
resulted in more comments and critical feedback, something we used to guide our creation and
presentation of prototypes [25].
19
2.5.1
User feedback on prototypes
We have already touched upon some of the differences designers may experience in
working in emerging markets - this extends to the use of prototypes as well. Maunder, Marsden,
Gruijters et al. discuss how users may not understand or may misinterpret low fidelity or
otherwise abstracted prototypes [9]. However, high fidelity prototypes may cause other
problems, such as the users giving feedback regarding the prototype rather than the actual
concept, in addition to the added time and cost of creating a higher fidelity prototype [5] [19].
Tao similarly found that users had difficulty in thinking abstractly about non-functional
prototypes [10]. However, he found that having a functional prototype helped spark
conversation and prove the commitment of the team to the project, especially since there already
existed skepticism from the failure of other university student teams. Perhaps most importantly,
he found that observing the user interact with the prototype revealed latent needs that would not
have been apparent from only conversation. He also suggests developing a functional prototype
that can be tested in the field. For our case studies, this was not a feasible option since the design
concepts were still in their early stages, so designing prototypes to convey product concepts was
an interesting challenge in both studies.
2.6
Case study methodologies
This thesis draws upon the methods outlined by Yin in analyzing the case studies [27]. There are
-
many examples of using case studies in the literature to examine design for emerging markets
such as Nieusma and Riley's study of a university collaboration between the US and Nicaragua,
and an NGO's work in Sri Lanka, or Tschudy, Dykstra-Erickson, and Holloway's study of using
a card based tool in rural India [14] [21].
2.7
What is the research gap?
There is as of yet limited research on of designing for emerging markets. Much research on
emerging markets focuses on specific technologies rather than the design methods employed or
strategies for success. While many papers on design for emerging markets emphasize the
importance of understanding users, outside of HCI, few of them discuss how well conventional
methods work (or fail), or examine the user research process itself. This is especially true for the
early stage of design, before there is a functional prototype.
20
In this thesis, we examine the effectiveness of different user research methods when applied
to the developing world. We also examine the role prototypes play in this research.
21
3
Case Studies
3.1
3.1.1
Methods
Why Case Studies?
To gain a better understanding of how the user centered design process can be adapted to the
developing world, and to test these hypotheses, we conducted two case studies. Case studies
allow us to study the user research process in a real world setting with a real world project. This
results in in-depth qualitative data that goes beyond the theoretical.
The first case study was an exploration of possible applications of a nascent solar energy
technology across India. The second case study was of a new product to help farmers in rural
India.
3.1.2
Case Study Methods
The case studies in this thesis were analyzed using an inductive strategy of searching for
emergent patterns in the user feedback [27].
3.1.3
Defining prototypes
3.1.3.1 Axes of Prototyping
Conveying the design concept to the users is a critical part in getting user feedback, and as such,
we were interested in understanding how to use prototypes to best convey our concepts.
Maunder and Marsden suggested that both low and high fidelity prototypes had their own
shortcomings in conveying design concepts [9]. We wanted to explore this idea further,
considering two specific aspects of prototypes. In addition to the dimensions of prototypes
explored in the literature, we define two more axes for the purposes of categorizing the
prototypes used in this study: the dimensionality and the level of narrative.
Dimensionality: In HCI, the closest equivalent is the use of paper prototyping versus actual
software. Since our case studies focus on physical products, we looked at using two-dimensional
drawings as well as three-dimensional low visual fidelity mockups.
22
Narrative: The other axis that was relevant to our work was the level of "narrative". The study
of this parameter was the result of a discussion with Subarna Basnet, an MIT PhD student and
native Nepali who helped advise our prototype creation [28]. His hypothesis based on his own
experience was that users would have a better understanding of a product if we visually
demonstrated how the product would be used or otherwise fit into their routine rather than just
showing a single image or prop of the object and verbally describing how it would be used.
3.1.3.2 Hypotheses:
One of the research questions we wanted to answer was how the type of prototype used affects
feedback. We focused on these two axes of prototyping, since they were what most
distinguished our types of prototypes. We had several hypotheses about this, influenced by
literature in HCI.
1) 2-dimensional prototypes would help avoid user fixation on particular features or form
factors, and therefore make it easier for the user to view the product as an early stage
concept with a variety of possible form factors or uses more than a 3-dimensional
prototype would
2) 3-dimensional prototypes would give users a more realistic understanding of important
physical aspects of the product such as size or weight than a 2-dimensional prototype
would, and as a result, would lead to a greater amount of and more specific critical
feedback
3) Using a narrative form would give the users a clearer understanding of the product use
case and how the product would fit into their daily routine
Based on the better understanding, lack of fixation, etc., the user would give better informed,
more well thought out, and more realistic feedback. In other words, the data we were looking for
to validate our hypotheses was a qualitative analysis of the feedback we got from users.
3.1.3.3
Types of Prototypes Used in Our Studies:
Over the course of the research, we used several different types of prototypes in several different
ways. As we describe each prototype, we will categorize them into the following framework,
based on the axes described earlier:
23
0
0
2-Dimensional
3-Dimensional
Dimensionality
Figure 3-1: Categorization of prototype by dimensionality and
narrative level
The way we categorize the prototypes along these axes is somewhat relative. For instance, we
would categorize a 3D mockup in which the designer role-plays using the product as having a
higher level of narrative than a designer who simply shows the user a 3D mockup and verbally
describes some ways it can be used.
3.2
3.2.1
Case Study 1
Background
This first case study looks at a project to find applications for a novel solar technology in India.
The technology is a solar thermal fuel composed of azobenzene-functionalized nanotubes and is
currently in development by Professor Jeffrey Grossman in the MIT Department of Material
Science and Engineering. The solar thermal fuel (STF) is a chemical that can, theoretically,
absorb solar radiation, store the energy for a number of days without increasing in temperature,
and when triggered, release the heat. Previously explored solar thermal fuels degrade after 20-50
cycles; however, the current STF being studied was tested and proven to be able to cycle over
2000 times with no loss of capacity, making the material potentially indefinitely renewable. The
advantage of the STF over similar heat storing technologies such as phase change materials is
that it does not need to be insulated, since the heat energy is stored chemically rather than
24
thermally. However, at this time, the STF is still in the experimental and research stage; a STF
that can meet the baseline theoretical specifications has yet to be synthesized, and a sample size
of STF that could be tested in the field has not yet been created.
Even with the newness of the technology, the MIT Tata Center for Technology and
Design wanted to begin work on understanding how this technology might be "productized" in
India. The initial application suggested was a solar cooker, which could be used to replace
traditional biomass, kerosene, and LPG stoves.
The most natural approach seemed to be to look at the problem from both a technology
push and market pull perspective. In practice, this meant identifying a number of possible
applications given what was estimated to be the performance envelope of the STF.
The author approached the problem from a user-centric point of view by considering user
needs in a variety of possible residential and commercial applications of the STF where heat
and/or energy might be required. This is where fieldwork in India was critical. The groundwork
was already laid when the author joined the project from a prior trip to India, which mainly
involved expert user interviews. For these studies we used several prototypes - two types of 2D
prototypes (storyboards and non-narrative images), and a 3D prop with very little narrative
element in its presentation.
3.2.2
Problem Identification
Although design thrust was largely governed by the technology, the reason that the Tata Center
wanted to use the technology in India was because of an already identified and well established
problem: burning biomass for cooking and household heating. Nearly 3 billion people
worldwide cook and heat their homes by burning inefficient biomass such as wood and animal
dung [29]. Measurements of carbon monoxide and of particulate matter in households as a result
of traditional cookstoves have been measured and demonstrated to very high, emphasizing the
need for alternative solutions to biomass [30]. The resulting illness caused by indoor air
pollution leads to over 3.5 million deaths a year - more than AIDS and malaria combined [31].
Solar energy is an attractive substitute to using biomass; as yet, there has not been a single
successful, widely adopted solar cooker that has replaced current cooking methods. It was hoped
that the STF could overcome the drawbacks of these other technologies and thereby decrease the
use of biomass.
25
3.2.3
Challenges of eliciting user feedback
In order to determine an appropriate use for the solar thermal fuel, we chose to obtain user
feedback on early stage design concepts, likely with little functionality. Additionally, many of
our intended end-users would have had little education and low exposure to technology, so we
needed to be able to convey the product concept in a way that would be clear enough to them to
get usable feedback.
3.2.3.1
Technology pull versus pull; approaching both simultaneously
By simultaneously considering the technology and the users, we hoped to match the capabilities
of the solar thermal fuel to an existing need. Although technology push can often fail, there is
some argument to be made for a push with a new technology [2]. Norman and Verganti argue
that human centered design, with a focus on the iterative process, is ideal for incremental design
[32]. However, radical innovation is achieved through a technology push or a change in
meaning. The Demand-Driven Innovation handbook similarly recommends mixing pull and
push if there is not already a demand, since it can be difficult to get communities to adopt new
technology [2].
The application space was explored initially through a combination of expert interviews
and informal discussions during early trips to India, discussions with members of the MIT
community, and online research of existing products that use heat. The applications chosen to
focus on were cooking, heating, ironing, hand heating, and produce drying.
3.2.3.2
Identifying Needs based on use analysis
User analysis was conducted through a variety of methods, mainly consisting of observations,
semi-formal interviews, and focus groups.
In order to observe the needs of a broad spectrum of users, we conducted field research in
three starkly different locations in India: Ladakh, Jaipur, and Delhi. In Ladakh and Jaipur, we
did a combination of observations and semi-structured interviews. In Delhi, we conducted two
focus groups.
26
User Methods
Socioeconomic
Status
Literacy
Rural/Urban
Exposure to
Solar Tech.
Deep hanging out,
process analysis,
interviews
Middle class
Interviews, informal
observations
Focus groups
Lower and middle class
Middle Class
Mixed
Mostly illiterate
Literate
Mainly rural
Mixed
Urban
High
I
Delhi
Jaipur
Ladakh ~
Unknown
Low
I
I
Table 3-1: Comparison of the locations visited
3.2.3.3 Ladakh
Ladakh was selected as an initial location to focus on for the solar thermal fuel for a number of
reasons. Although Ladakh is generally not representative of the rest of Indian culture, it is a
good beachhead market, since the climate and culture are primed for solar technology. The
climate in Ladakh is sunny and clear most of the year, and it has some of the highest solar
insolation in the world [33]. India's Ministry of New and Renewable Energy (MNRE), via the
Ladakh Renewable Energy Development Agency (LREDA), has made a strong push for solar
technology in the region for those reasons.
The summer of 2014, we spent four weeks in Ladakh to do field studies. Before
beginning with the field studies, we conducted eight unstructured interviews with experts to gain
context into the state of solar energy and initial ideas regarding the solar thermal fuel. These
experts were employees of NGO's, members of LREDA and the MNRE (Ministry of New and
Renewable Energy), and local entrepreneurs.
Our field studies began with a three-day trip with one interviewer and a translator, using
ethnographic methods of deep hanging out, process analysis, and semi-structured interviews.
We spent two overnight stays in different of rural middle class farmers households where we
began the first few hours simply by observing, then doing a process analysis of users cooking,
and eventually ending with a semi-structured interview. In between the two overnight stays, we
also spent an hour doing another deep hang out at another rural middle class household.
Observations and interview notes were documented in hand written notes, and some interviews
were recorded with permission of the users. Semi-structural interviews were guided by a list of
questions. The interviews focused on questions about current lifestyle, and began by asking
27
basic lifestyle questions ("Where do you work?", "Can you tell me what you did yesterday?"),
then about cooking habits, household heating, and use of solar hot water heating ("How many
meals did you cook yesterday?", "What do you like about cooking?", "How do you heat your
house in the winter?"),. Most of these were with people who spoke English, which eliminated
any difficulties with translation.
Figure 3-2: User in Ladakh demonstrating a solar cooker
The observations were occasionally supplemented by asking clarifying questions of either
the people living in that household or the translator. These initial observations helped form a
basis to understand the context, lifestyle, and potential needs to focus on, as well as some
quantitative data such as the amount of fuel people were using. After the initial round of
observations, we revised portions of the interview to fit what we observed.
After the initial three-day trip, we were joined by an MIT student who assisted with the
field studies by note taking or interviewing users. We then performed a mix of further hang outs
and semi-structured interviews, where we added the storyboards and more speculative questions
to elicit opinions on the solar thermal fuel concepts, in addition to asking questions just about
current lifestyle. Seven of these interviews were conducted during a four day period traveling
through the farming villages of Yangthang, Rizong Monastery, Temisgang, Hemis and
28
Lamayuru, and another two where conducted in Choglamsar, an urban village near Leh. Each
interview lasted approximately 30-60 minutes, with a translator when necessary. Several of
them were conducted in small groups.
We had several storyboard pages, illustrated in a simple cartoonish manner.
0
Figure 3-3: Storyboard used to demonstrate solar thermal fuel as a portable device
The storyboards were drawn in this simple manner to avoid distracting details or unintended
cultural differences, following the guidelines suggested by Truong, Hayes, and Abowd, as well
as practical advice given by Subarna Basnet a Nepalese PhD student at MIT, and Isaac Gergan, a
Ladakhi artist who had experience using storyboards to present concepts to Ladakhi farmers
[28][34][35].
The first two pages show two different ways this new technology could be use (either as a
portable system, or a stationary roof mounted system) without specifying the application for the
product. The next two pages show several options for cooking and heating. The last page shows
miscellaneous other applications (ironing, hand warming, and agricultural drying). The first two
pages described a story of the product being used in multiple panels, while the actual
applications generally are demonstrated as single panel images.
We conducted the first part of the interview as before; then for the second part we
introduced the idea of the new technology, and in conjunction with the storyboards, asked more
29
freeform questions about the technology, such as what they thought about it, etc. The
storyboards were generally presented in an "A, B or C" fashion in which the interviewees were
shown the storyboards and asked to select between different options.
In addition to storyboards, we also built a works-like prototype of one potential form a
product based around the STF might take. This mockup, nicknamed the "magic box", was a box
containing a hot plate and battery, roughly the size and weight of what the product might look
like for the current best case scenario for the STF. The "magic box" was used during several
interviews (one in Ladakh, three in Jaipur) to supplement the storyboards. We told users that it
was a model of a solar device that would have to be charged outdoors, but could be moved
indoors, and could produce heat. We suggested applications to them such as cooking and
household heating, but tried to make it clear that the "magic box" was not designed with a
specific application, but meant to demonstration overall what a possible physical model might
look and work like.
Figure 3-4: The "magic box" physical prototype
30
In addition to interviewing households in rural areas and in a few urban areas, we also
interviewed several hotel owners in Leh, since the tourism industry plays a very large role there.
These were more informal interviews, generally asking about hot water heating, space heating,
and what they thought of the new technology, and were not considered in the analysis for user
feedback since we were more interested in studies in which we asked users to evaluate the
product concepts.
3.2.3.4 Jaipur
Rajasthan is another area of India where there is a lot of solar insolation and a large number of
solar technology projects. However, unlike Ladakh, the climate is quite hot, meaning that
household heating, one of the primary applications identified, is not appropriate. The culture is
also more similar to the rest of India than Ladakh.
We spent only a few days in Jaipur, where we aimed to conduct interviews in a similar
way. However, there were many difficulties with the interviews in Jaipur due to the translator,
which will be described later. Despite that, we were able to use these opportunities to observe the
lifestyle and get some useful feedback.
We conducted group interviews rather than one on one. This was more of a necessity
than with the intent of having group interviews. It can be difficult to isolate a single person for
an interview, since interviews in rural and semi-rural areas were largely conducted outside, and
other neighbors would come over to join in the discussion. The result is that even if we started
with a smaller number of people by the end the group size had usually inflated. Additionally, the
translator took our desire to do "10-20 interviews" to mean the number of people rather than
interviews, so our first "interview" was with a group of about 10 men. Between the large
number of people and the language barrier, it became very difficult to hold a conversation.
Additionally, large groups of children from around the village tended to follow us around and
interfere with the interview.
We interviewed four lower and middle class groups in rural and semi-rural villages (no
one we interviewed was severely impoverished), who mainly earned their living from farming
and cattle. The last day we interviewed two urban middle class households. Each interview
lasted between thirty to sixty minutes. For the first interview, we used only the storyboards, but
for all the rest we used the "magic box" as well. However, we found it was difficult to explain
because of the translator and other external factors such as crowds of curious children.
31
Figure 3-5: Interview group, onlookers, and translator with the "magic box"
3.2.3.5 Delhi
Delhi was selected as a location so we could get a sense of what the urban middle class thought
of the solar thermal fuel.
In Delhi, we conducted two professionally led focus groups with Susmita Misra, a local
marketing specialist. They were done in a more traditional marketing-style focus group. The
first group consisted of eight upper middle class, educated, urban women. The second group
consisted of eight middle class and slightly less educated urban women. The first part of the
discussion was questions more about things such as their family, what they do in their free time,
etc., to get the women more comfortable and to gain context into their lifestyle. Then they were
asked questions about budget concerns, etc. After a number of these types of questions, Susmita
introduce the idea of the new technology. People were asked for feedback. They were asked to
create a collage of potential ideas for applications and describe them.
Unlike in the other locations, we did not use any type of prototype. Susmita instead
described the solar thermal fuel as "something which store the solar energy and then convert it in
to heat, which we can heat things with it, and we can set it inside or outside the kitchen", and
then went further on to describe a portable and a roof mounted version of the product. This
32
generated a number of follow up questions from the participants to try to get a clearer idea of the
technology. However, unlike the previous studies, rather than asking the participants to choose
from a set of concepts, she asked them to brainstorm applications.
3.2.4
Categorizing the Types of Prototypes used
For the studies in Ladakh and Jaipur we used the following types of representations:
Storyboards (solarcollection storyboards: solar collector form factors)
Non-narrative images (STF applications: single and multi-panel form factors for specific
applications)
3D Mockup ("magic box ")
Figure 3-6: Examples of non-narrative images. Left: ways the STF can be used for cooking. Right:
Potential form factors STF-based heating.
They fall along the graph in the following way:
33
Solar Collection
Storyboards
0
STF Applications
0
2-Dimensional
3-Dimensional
Dimensionality
Figure 3-7: Classification of the Solar Collection Storyboards, the STF Applications, and the "Magic
Box"
The reason we classify the magic box as being low on the narrative scale has less to do with
anything inherent to the prototype itself and more to do with how we used it. We did not role
play or act out with the model to demonstrate its use case, but rather showed it to users and listed
to them how it could be use as a portable system that would be charged outdoors, taken inside,
and used to produce heat.
3.2.5
3.2.5.1
Case Study 1 Discussion
User Analysis Methods
The table below summarizes the different methods used in the first case study. Some of the
categories are further elaborated on below. Our observations in using these methods generally
aligned with what was outlined in Smith and Leith's D-Lab Scale-Ups User Research
Framework, as well as what Courage and Baxter described as what to expect and be wary of for
different methods [18].
34
Type of Info
Number of sessions,
people per session,
time per session
Advantages
Disadvantages
General lifestyle;
cooking process
5 sessions
Get a good
understanding of the
Can be confusing
with just
1-5 people
lifestyle; get to see
latent needs; find
inconsistencies
between what people
say and do; less
obtrusive
observations to
understand what is
going on sometimes;
time consuming
Time consuming
o.
30 min - 1 hour
E
W
.
*
'~
General lifestyle,
estimates of numbers
for use of current
products/methods
(i.e., amount of fuel
used), details on
lifestyle during other
times of the year
(i.e., winter heating);
likes/dislikes about
current products or
processes; concerns
5 sessions
30 min - 1 hour;
overnight
Get a good
understanding of the
lifestyles which
makes it easier to
understand what
questions to ask
Everything above;
10 sessions
Storyboards helped
Requires a translator
clarify how STF
with good
1-2 people,
occasionally large
could be used;
allowed people to
understanding of
STF; no way to
groups (6-15);
choose between
know if users are
options; gave people
an analogy to
something they
might already know
making incorrect
assumptions about
the STF based on the
images
"Magic box" opened
up the conversation
to more critical
questions
Requires a translator
with good
understanding of
STF; people became
fixated on "Magic
Box" as a cooker
exclusively, as well
as fixating on
unimportant details
1 or 2 people
opinions and
i
questions about the
T STF
30 min - 1 hour
0
Everything above;
questions very
specific to the
prototype
4 sessions
1-2, occasionally
more
30 min - I hour
-_ _
_of
the prop
35
General lifestyle;
concerns; opinions
about uses of current
products/processes;
feedback, criticisms,
and ideas about the
STF
2 sessions
8 people per session
About 2-3 hours
Get a wealth of
information very
quickly; people
bounced ideas off
each other; people
are more open
because group
setting
No way to verify if
what users say is
actually what they do
at home; more
difficult to imagine
design parameters
based on lifestyles,
homes, etc.;
possibility of
groupthink/one
person leading the
__
_
_conversation
Table 3-2: Comparison of different user research methods
Reception to 2D Prototypes (storyboards):
The storyboards proved to be a useful tool to clarify the idea of how this technology could be
used. We found people were able to understand the storyboards, and generally seemed to be able
to extrapolate that the product did not need to look exactly like what was depicted, but could
function similarly. Drawings are known for their ability to preserve ambiguity, thus leaving
room for the viewer to interpret meaning, something that was both an advantage and
disadvantage in our studies [36].
With some of the images, such as the heaters, the drawings analogized to existing
products. The reason for drawing these form factors was because that early in the concept stage,
we did not have a clear idea of exactly what would be the most efficient way to transfer heat
from the solar thermal fuel to a room, but imagined that it could work similar to a heating
element or a heated fluid in existing heater types. Therefore, we envisioned these heater
concepts to look like and work like existing models.
Because of the similarity to existing products, it gave some people an easier time
understanding not just how the product would look, but also how the appliance could function.
However, in one case, a user chose one form factor by default because he hadn't seen form
factors of the other types, and felt he could not make an informed decision. For likely similar
reasons, some users simply said they thought any option for the heater would be fine. There may
be a better way to convey the idea without comparing it to existing products, or by clearly
36
identifying the trade offs of each choice (i.e., a smaller heater is more portable but produces less
heat).
In addition, one advantage of showing multiple concepts was that the feedback was more
critical than we expected, because people were forced to choose between options, giving us an
idea of relative preferences (even if we do not know ground truth). We believe that if we had
presented users with a single idea, people would have been more inclined to say they liked the
idea but with the storyboards, they were at least forced to choose a preference. Our observations
in showing people multiple concepts matched with the observations of Tohidi, Buxton, Baecker,
et al., who found that users were more reluctant to criticize ideas when presented with a single
design than they were when presented with multiple ones [25].
One unexpected problem with the two dimensional prototypes was that people made
automatic assumptions that we did not expect. One such example that we discovered after
talking to our translator was that he mentioned that people assumed the stationary solar
collecting system is bigger, therefore better but more expensive. Although this particular
assumption will likely be true, it's difficult for us to know what other implicit assumptions
people were making that might have been erroneous but swayed their decisions that they did not
tell us when justifying their choice.
3D prototypes: Reception to the Magic Box
Before going to India, we were concerned that users would get distracted in the details of the
box, and not be able to abstract the idea to other form factors. However, using it in Ladakh,
these were not issues. The group we interviewed at first did question specific details about the
form factor such as whether or not we could add a stand, or whether the heating pads had to be
two distinct units, but when explained (via a translator) that the form factor could change, they
readily understood. However, they still viewed the magic box only as a cooker rather than a
stand-in for many possible devices. The advantage of presenting the physical box was that it
gave something for people to touch and point at, and the questions they asked seemed more like
the types of questions people would ask if about to buy a real, new product. It caused them to
ask a number of critical questions that would not have arose with only a storyboard, and these
questions helped make it apparent that the cooker was not a good application for those people,
and helped us then move on to other topics such as heating. In that sense, it allowed people to be
37
indirectly critical. For instance, they asked a number of questions about how long one could
cook with the stove, how many cups of tea they could make, and how long it would take, and
then would compare it to the performance of their gas stove (which was higher performing in the
metrics they were questioning); eventually through enough of these questions with answers they
were not impressed by, they decided that heating makes far more sense.
Design the fact that users saw it only as a cooker at first, we were able to move smoothly
from using the magic box to use the storyboards and have people think more critically about
applications beyond cooking. Strangely, they were not as interested in a live demo or actually
trying it, perhaps because they knew it was not the final product.
In Jaipur, the Magic Box did not prove to be as useful, though this is attributed largely
due to difficulties with our translator. However, it did make at least some things clearer to the
translator who had misunderstood the initial concept. On the last day, when we were in urban
households, the box became useful as a talking point.
Focus Groups:
We found that the first group, consisting of better educated and higher economic class women
than the second group seemed to think more seriously and critically about the technology. The
second group, were more willing to accept the technology and did not ask questions that were as
probing.
Because it was a group setting, people bounced ideas off each other, and also seemed
more willing to be critical. This could be partially because Susmita, a local Indian, was
moderating, which may have made people more comfortable with responding freely, which
matches the observations made in the literature by Vatrapu and Perez-Quifiones [17]. Another
reason was because once one person voiced a concern, other people started to think more deeply
about the technology.
The focus groups had several advantages and disadvantages over individual interviews.
The focus groups allowed us to get information far more quickly and efficiently than we did with
individual interviews.
One of the major disadvantages of the focus group was not being able to observe these
women's lifestyles for ourselves. One complication there is that while women may say one thing
(especially in a group of peers), they may actually be acting differently at home. Also important
38
for the designer, not being able to see the kitchen set up or household makes it a lot more
difficult to envision the possible form factor and design parameters of a future product.
For the focus groups, no product representations were used. Participants relied entirely
on Susmita's description of the solar thermal fuel. Despite that, the first group in particular was
able to understand it enough to the degree of brainstorming applications. In fact, the first group
understood the concept enough to think critically about some of the failure modes, such as the
fact that it would be expensive to install a piped system in an apartment building. However,
there were still a number of participants who remained unclear on the details of the concept, and
would get hung up on things such as the number of rooms a solar thermal fuel heater could heat.
There was also no way to verify that what the participants were envisioning was in fact the
concept we were trying to convey other than by the types of responses they gave.
We saw that the use of 2D prototypes helped encourage users to think outside of the
"magic box" of the solar thermal fuel - taking it further and providing only verbal description
meant that participants could interpret the technology even more ambiguously, and as a result,
thought of all sorts of applications for the solar thermal fuel by thinking about it very broadly in
terms of where heat could be used. Of course, as Courage and Baxter warned, they did not know
or understand the nuances of the technical aspects of the project, so many of the ideas they had
would not have been feasible - but they used the same thought process we had used to initially
generate the application space and every single one of their suggestions but one had been
something we had also thought of [6].
The question arises, then, for something still early stage such as the STF, would it have
been better to use no prototype at all? Would non-educated, rural users with lower exposure to
technology been able to imagine the STF as well as these women had? Unfortunately, that was
not something we had an opportunity to test.
39
3.3
Case Study 2
3.3.1
Introduction
This second case study looks at an ongoing project in the Tata Center being conducted jointly by
Dr. Chintan Vaishnav, Prof. Anastasios Hart, and graduate students Soumya Braganza and Ron
Rosenberg. They are developing a new soil testing technology for Indian farmers to be able to
better analyze their soil and identify the type and amount of fertilizer required.
During the winter of 2015, we traveled to Hubli, India and worked with the Deshpande
Center. The goal was to better understand users' perceptions towards soil testing, and getting
early user feedback on their concepts. While the project in the first case study was focused on
finding a suitable application for the solar thermal fuel and getting an idea of what product
specifications might look like, this project already had a clear vision of the problem they were
aiming to solve and a potential concept. Because of that, they were more focused on getting user
feedback for the concept, and defining the product specifications that would arise from user
needs.
3.3.2
3.3.2.1
Process
Interviews
Before conducting interviews with users, we first had the opportunity to interview a government
employee who worked in a lab processing soil samples. Then, we interviewed a number of small
acreage farmers in villages near Hubli. Interviews were conducted primarily with groups of
people, for the same reason as were necessitated in Jaipur.
For each of these interviews, we had one person from our group asking the interview
questions, one person taking notes, and one person taking photos (these interviews were not
recorded). We had one or two translators from the Deshpande Center to translate the discussion.
Two translators were not actually required (and in fact, generated confusion at first when both of
them would try talking over each other), but the initial plan had been to interview two groups in
parallel. However, what happened in practice was that interviews were conducted one at a time.
For this reason, we had an excess number of people, which if the interviews had not been
"a
s a g"up pJobaly VVUoU have served Lo 111LIMiLdt te interviewee.
We conducted five such group interviews, each with five to ten people. We had three
interviews with only men, and two with only women.
40
Figure 3-8: Conducting group interviews in a farming village near Hubli
3.3.2.2
Workshop
We conducted three workshops to evaluate concepts for soil testing. The workshop was located
on the premises of the Deshpande Center's agriculture school outside of Hubli, which made it
relatively accessible to villagers. Also, the participants were all already familiar with the
Deshpande Center, so they felt relatively comfortable on the premises. The Deshpande Center
was also responsible for recruiting the farmers.
We did two workshops on the first day, one in the morning and one in the afternoon, and
a third workshop on the second day. The workshops lasted roughly three or four hours,
including a tea break, except for the second one in which we only had an hour and a half.
Although we aimed to have between 8-10 farmers per workshop, the first workshop had
26 participants to begin with (some left early, but by the end we had14 men and 4 women), the
second had twenty men, and the third had twelve men and six women. The disparity in numbers
results from the Desphande Center over-inviting farmers, anticipating that only half of those
invited would attend. However, that turned out not to be the case.
41
The farmers all spoke Kannada as the primary language. Since none of the members of
our team spoke the language, one of the Deshpande Center volunteers acted as the moderator.
We reviewed the activities with him beforehand, and a member from the team was on hand to
provide instruction if necessary. We paired volunteers from the Deshpande Center with note
takers from our team. During activities involving the entire group of farmers, our team and the
translators sat in the back and quietly translated. When we broke out into subsections, a note
taker and translator pair joined each section.
The workshop was broken down into several parts, with variations on each of those parts
during the different workshops.
1) Introductions and Project Purpose
2) Pain Points
3) Product Attribute Exercise
4) Soil Collection Methods Exercise
5) Soil Card Testing Exercise
6) Open Discussion/Suggestions for Improvement
Each section is discussed in further detail below.
The order of the exercises was selected for two reasons. First we wanted to warm up the
group, so initial exercises were more casual discussions before moving on to more structured
exercises. The most important exercises were left for last, for when participants would feel most
comfortable. The second reason for the order relate to the soil testing directly. These exercises
are presented in the order of operations in which actual soil collection and testing would occur.
3.3.2.2.1 Introductions and ProjectPurpose
First, we introduced ourselves, the MIT Tata Center, and gave a brief introduction to the
project. We kept the description of the project intentionally vague at this point so as not to lead
the conversation, limiting ourselves to saying that we are trying to get a better understanding of
soil health. Additionally, we attributed the project to the Tata Center itself, trying to
depersonalize the project so people would feel more comfortable criticizing it. We emphasized
that we were there to learn from them, that there are no right or wrong answers, we want their
honest opinions, and that criticisms help us. Thcn, we asked them to introduce themselves one
by one, by name and what type of crop they grow. In an effort to try to warm up the group, we
42
also asked them to name a favorite Bollywood movie or type of food (which we also did in our
introductions), which some participants did.
3.3.2.2.2 PainPoints
Much of the first part of the workshop was devoted to trying to warm people up,
including this next exercise. In this second section, we asked the participants to list some of the
difficulties they experience with their day-to-day life on the farm.
3.3.2.2.3 ProductAttribute Exercise
The following exercise was used to continue to warm the group up and to get them
comfortable with more structured exercises, but also to start to get some sense of what attributes
of a product users value. We did this exercise in three different ways, but the overall idea was to
ask users to identify from among a set of product attributes what they value the most and why.
As the designers of the soil testing technology did not yet have a range of expected values for
attributes of the technology such as cost, longevity, etc., they preferred to not use the soil
technology as the example. Instead, we asked people to choose a product they were familiar
with; since the most important attributes vary by product, we were more concerned with their
reasoning rather than the actual attribute.
Before the exercise, the participants were asked to discuss a product they valued on the
farm, and why they valued it. The idea behind this was to get people already in the mindset of
thinking about products, and to give the designers a sense of what matters to users. This initial
part was dropped during the third workshop, since we found it was not especially effective.
The three variations of the exercise were as follows:
1) Product attribute cards: This initial exercise was prepared before we arrived in India,
and were unsure about the literacy rate of the farmers. We decided to use five picture
cards depicting the attributes of cost, durability, reusability, time taken, and labor
intensiveness. The idea behind the cards was that we would describe out loud what they
were meant to represent, and that the image would help the participants remember what
attribute it represented. For the exercise, the workshop was divided into subgroups of
four to five people, and each person was given a set of the cards. They were told to think
of a product they used regularly on the farm. Then, they were given thirty seconds to
remove the attribute that mattered the least to them. This was repeated two more times
43
until they were left with only two cards, the idea being that with only thirty seconds, they
would choose based on instinct rather than overthinking the exercise. After that, they
were asked to describe what product they chose, and explain why they chose the
remaining attributes.
Figure 3-9: Product attribute cards depicting (from left to right) durability, time taken, and labor
intensiveness
2) Product attribute list: This exercise was designed during the break between the first
and second workshop. The designers decided to add a few more attributes for this
exercise: safety, simplicity, and accuracy. Then, all these attributes were written on the
board in English and then translated into Kannada. By this point it had been established
that the majority of the participants were literate in Kannada, and in addition, the
attributes were all read and described aloud, so the picture cards were deemed
unnecessary. Once again, the group was divided into subgroups of four to five people.
The moderator went through the list of attributes one by one, for each one giving a
concrete example. For instance, for "accuracy", he described the ability of a machine to
sow seeds in a straight line. Then, in the groups, the participants were asked to rank each
attribute's importance as "high", "medium", or "low", while thinking about a specific
product, and to justify why.
3) Product attribute trade-offs: This exercise was designed to understand the types of
trade-offs users are willing to make. The designers of the soil technology identified
several pairs of trade offs they anticipated they might have to make with the soil-sensing
product, such as cost versus durability and accuracy versus simplicity. For this exercise,
44
we decided to give the option of not choosing a specific product, but just describing the
attributes in general. The group once again divided into subgroups of four to five people.
Then, the moderator would explain the tradeoff pair, and give a concrete example. For
instance, in explaining the trade off of labor intensiveness versus time, the moderator
described a pesticide sprayer that is heavier and therefore more labor intensive to use, but
takes less time, versus a light one that takes a longer time. The users were then asked to
explain what product they chose, if any, and then to explain which trade off they would
make, and why. For this activity in particular, these instructions often took a long time
because the examples, difficulty with abstract ideas, and often were repeated again by the
translators in the subgroup.
Figure 3-10: Image from the first workshop
45
3.3.2.2.4 Soil Collection Methods
The purpose of this next exercise was to generate discussion about the different concepts for
collecting soil. Since a lot of the more concrete details such as cost of the method or time to
collect the soil were as yet unknown, the focus was less on trying to identify a single best
concept, and more on early on getting user feedback on the pros and cons of each method, as
well as any problems they anticipated. Therefore the exercise was structured as an open
discussion after demonstrating or describing the different methods. We prepared two types of
prototypes for this exercise, storyboards and physical props.
1) Storyboards: Four storyboards were drawn for this exercise, each of them depicting
how the soil would be collected. For each concept, the storyboard was projected and
then the moderator went through, panel by panel, and explained the method of soil
collection. For the first workshop, all four methods were described before the
discussion took place. For the second workshop, after each storyboard, there were a
few minutes of discussion.
46
Do this 15
ilifi
/
I- %
times
Figure 3-11: Storyboard used to demonstrate collecting soil for testing as it is currently practiced
2) Roleplay with props: A set of simple props including mock up products created out
of blue styrofoam and plastic tube were used to roleplay the soil collection method.
This was done in a garden outside of the room the workshop was conducted in. The
props were installed in the garden, and the moderator acted out how they would be
used. For the method that involved digging a hole, one of the users was invited to try
by actually digging a hole with a shovel provided. Then, after all four were described,
there was a moderated discussion.
47
Figure 3-12: The moderator and translators roleplaying with the physical props
3.3.2.2.5 Soil test cards
This was, for the designers, the most important part of the workshop. Mockups were created of
several variations of the soil test cards. Participants were divided into subgroups of four to five
once more. Each group was given a soil test card kit one at a time, and asked to "decode" the
card, and then evaluate the difficulty in decoding it, before being given the next one.
Participants were given basic instructions, but had to figure out for themselves how to decode the
card. Note takers observed users as they decoded each card.
3.3.2.2.6 Open Discussion
The last part of the workshop was a moderated discussion inviting feedback, ideas and
questions.
48
3.3.2.3
Categorizing the Types of Prototypes used
For the studies in Hubli, we used the following types of representations:
Storyboards (soil collection storyboards: methods of collecting soil)
Single panel images (product attribute cards: depictions to describe attributes of products
such as durability and cost)
3D Mockup (soil collection mockups: simple foam and PVC mockups to describe
methods of soil collection)
They fall along the graph in the following way:
JC
Solar Collection
Storyboards
Soil Collection
Storyboards
0
STF Applications
0
2-Dimensional
3-Dimensional
Dimensionality
Figure 3-13: Classification of the Soil Collection Storyboards and the Soil Collection Mock Ups
The reason we classify the soil collection mockups further along the narrative scale than
the "magic box" is because of the way the mockups were presented by actually roleplaying their
use. However, they did not go as far as the solar collection storyboards in showing the full story
of the product use case.
Soil cards: We do not include the soil cards in our categorization, since they present an
entirely different type of prototype. With all the other prototypes, they are low fidelity
49
representations of very early stage, not yet fully formed concepts. However, the soil cards are
much more fleshed out concepts that are actual looks-like and even work-like models in how the
users interpret the data (even if it is not functional in the sense of testing the soil). For these
reasons, we believe that they are not comparable to the other prototypes in our studies, which are
much earlier stage design concepts.
3.3.3
3.3.3.1
Case Study 2 Discussion
Product Attribute Exercise
Since product attributes are somewhat abstract when not pertaining to attributes of a specific
product, the participants had a lot of difficulty understanding the exercise. Additionally, because
of the way the discussion was set up and because the richest qualitative data came from how the
participants justified their choice, the translator-note taker system was really inadequate for this
set of exercises.
While the users had no difficulty understanding cost, other attributes, such as accuracy,
durability, laboriousness, were much more difficult to convey, especially when described in the
general and not applied to a specific product. Part of the difficulty was that we asked each user
either to envision their own product, or to try to think about these traits in the general. When we
asked them "in the general" and gave an example, people tended to fix on that example.
Another problem was that the attributes we chose were those relevant to the soil testing, but
not necessarily relevant to other products. For instance, accuracy of the results is potentially a
very critical attribute of a soil sensor; but to trying to use an example to substitute the soil sensor
can be confusing. The moderator gave the example of an expensive machine that deposits the
seeds accurately in a line, versus a cheaper machine that sometimes misses. The general
consensus was that people would prefer a more expensive but higher quality machine. In fact,
"quality" was the word more often used, and accuracy (as well as other attributes) generally just
fell under that umbrella in the participants' responses.
2d representation. During the first run of the exercise, we tried using picture cards to convey
the attributes, which wound up resulting in a misinterpretation of several images, both due to the
translator, and to the users' own interpretations of the images. In this case, being able to
ambiguously interpret an idea was both an advantage and disadvantage of the 2D representation.
In the first case study, the storyboards helped users envision the solar thermal fuel as a variety of
50
form factors. But in this case, it worked against us by confusing users. For instance, a clock,
meant to symbolize the attribute of "taking less time", was interpreted by some users to mean
"how long the product would last". Similarly, a picture of someone throwing something in a
trash can, with a red X, was meant to show reusability; however, it was taken more literally as
how reusable the product could be in terms of the life time of the product (i.e., one person said
that they thought reusability was important because they would expect to be able to "reuse" their
oxen many times). On one hand, this means the quantitative data from the different groups about
which picture cards they used is mostly useless, since individual groups and people interpreted
the pictures differently, but the qualitative data in which participants walked us through their
thought processes, was very rich.
This was one of the exercises where all of our versions of the exercise hinged on using the
picture cards. Given the confusion caused by them, our attempt to quickly revise the exercise
during the hour between the two workshops resulted in an activity that was not fully fleshed out.
In particular, by asking the participants to score each attribute "low", "medium", or "high",
without asking them to rank the attributes relative to another, resulted in a somewhat loaded
question, and in one subgroup, the participants simply ranked most of the attributes as "high".
Interestingly, this did not occur in all the subgroups, and once again, a lot of rich qualitative data
was gained from the explanations provided. However, it was even more time consuming than
the previous iteration of the exercise. Moreover, at least one participant was so confused by the
activity that he voluntarily withdrew from the activity part of the way through.
The third version of the exercise had its own difficulties. Many people did not understand
the concept of trade offs. For instance, for the trade off of labor intensive versus time taken to
complete the task, the moderator used the example of a pesticide sprayer that is heavier but takes
less time to use, versus a lighter one that takes more time. One group said they would hire more
laborers, since time is of the essence. It is unclear whether the translator misinterpreted "labor
intensive" as "laborers", but in any case there was more difficulty in understanding trade-offs
that did not involve cost.
Ultimately, despite the misinterpretations and confusion of these abstracted attributes, we
still got a lot of insight into what people from when they explained their rationales for all three
activities. For instance, one thing that emerged very clearly for the designers was that having the
sensor be low cost was often at the bottom of people's priority list. Generally, participants
51
preferred to invest in quality even at a higher cost, except for items where the quality didn't
matter as much. This is consistent with observations by Austin-Breneman and Yang [20]. For
instance, one participant stated that he would only buy a tractor manufactured by Mahindra, a
well-known company, even though it was more expensive, since the brand name implied quality.
However, he didn't care about the brand or quality of simple farm tools, since he would have to
replace them every few years anyway.
Ultimately, designers should be clearly aware of how the tool or method they are using is
affecting the data, and know that sometimes the quantitative data may be misleading. Moreover,
the quality of the data from these exercises in particular was very dependent on the translator.
3.3.3.2 Soil Collection Methods
One of the things we were very curious about studying further from our experience in Ladakh
was how people responded to the storyboards over the props. As mentioned earlier, we had
hypothesized that the storyboards would allow participants to envision the product more
holistically, while the mockups might give them a clearer understanding but lead them to fixate
on irrelevant details. What actually happened in Hubli was exactly the opposite.
The storyboards had a few failure modes. The first was that the way they were presented
was very important. The first time around the translator gave far too much detail, as though he
was instructing them step by step how to do the soil test. In fact, he went way off the script and
off the one page story described; the result was that people got entirely fixated on the details and
missed the point. For instance, they began asking questions about the specifics of the placement
and shape of the holes dug to test the soils, something that was not mentioned in the storyboard
and not relevant to the questions the designers needed answered.
The second time around, the moderator presented the storyboards far more briefly.
However, even then, the storyboards failed to give a sense of how difficult the work would be
(i.e. that digging holes is much more difficult than collecting water from a wicking device). Not
only that, it wasn't clear to us that the participants understood each of the design concepts.
In contrast, the props gave a very clear idea of how each product functioned, and the
amount of labor required to each soil collection method, so we think feedback from that was
Mnre realisctic.
Interestingly, we got about the same amount of criticism with the props than we did with
the second trial of the storyboards. This may have had more to do with the group, since the
52
several of the other people involved in the workshop thought that the second group of people was
overall "more negative". We also got more questions with the props than without.
3.3.3.3
Soil Card testing
The soil card testing was the most successful in engaging the participants. The game-like aspect
of it was so appealing, that most of the participants, upon completing each card, often were eager
to immediately move on to the next one. We found generally that participants were far more
interested in "doing" than simply looking at and commenting. It also provided the most specific
feedback to the designers about the design they had created.
3.4
Limitations
There are limitations to doing case studies following real world field studies. The first is that, for
such a project as these ones, the sample size is relatively small, so the qualitative data must be
depended on more than the quantitative. The second drawback is that it is very difficult to set
up a controlled experiment in a real world context, particularly when the primary focus of each
case study was to gain insight into user needs for a specific project by whatever means
necessary, rather than focusing specifically on the questions we wanted to answer. The third is
that we are working with actual people, and things can be messy and unclear. Therefore, the
results of each case study are not always generalizable, but often very nuanced and rich.
53
4
4.1
Results
Classifying Types of Feedback
In order to understand our results and how they relate to our initial research questions, we need
to first classify the type of feedback we received. For this analysis, we look at the comments
gathered specifically in response to the designer presenting and asking about novel product
concepts with consumers. This is different than earlier stage work in user needs assessment
without a specific concept in mind (such as during the initial context gathering observations and
interviews). Therefore, for this analysis, we focus on interviews or portions of the interviews,
workshops, and focus groups where we asked specifically about the product concept.
Type
Preference
Definition
User ranks one
concept or
attribute over
another
Non-specific
User gives
These types of comments were generally
"Ifcompleted, the
positive
positive feedback
not useful, although they sometimes gave
project would be very
but does not rank
or express
us an idea of how honest the user was
being. We sometimes got non-specific
nice", "Very good,
good idea",
Value to Designer
This was generally useful to give us a
sense of how users felt about one concept
over others. As with any of the
feedback, there is no real way to know
how honest or reliable the answers are
for certain, so even if a user expresses
preference for option A over option B, it
might be in reality they don't like either
but for a number of reasons (such as
politeness), do not feel comfortable
saying so. Additionally, a useful
assessment of one concept relative to
another depends on a clear understanding
of the product, which we found
sometimes users did not have. It is a
matter of further research to understand
how much to take user responses at face
value; we found at times it was very
obvious when the user misunderstood a
concept, but still provided valuable
feedback by providing justification for
their choices, but other times, we were
unsure if the user was really making an
informed decision. Even subtler is when
the user infers something from the
prototype that was not intended, and the
Examples
"Room heating is
better than cooking",
"A solar room heater
would be the best
use", "Cost is more
important to them
than time", "The first
two are easier, since
the result will come
sooner"
designer does not know that.
54
preference for one
concept above
others, or give any
reason
positive comments when a user did not
understand a product, but seemed to want
to please us. On more than one occasion,
users would ask a number of critical
questions that made it clear that the
concept did not meet their needs, but
then closed the conversation with a
generic praise.
"Everything is fine"
Specific
User gives
Generally, positive feedback was only
"It's good that it is a
positive
positive feedback
useful when the user coupled their
one time expense",
with reasoning
compliment with a justification for why
they liked a certain product concept,
"We would save
electricity", "This
because then it both demonstrated an
would save money",
understanding of the concept (making the
"I like that the testing
comment more reliable), and gave a
sense of what mattered to users.
This was one of the hardest type of
feedback to get, but when we got it, it
was very useful.
is at the source"
Criticism
User criticizes the
concept, e.g. by
identifying
problems with or
saying they would
not want the
product
Questions
User asks a
clarifying question
of the concept that
Users sometimes asked a number of
specific and critical questions that helped
reveal people's concerns, needs, and
"I don't want to
replace anything in
my home with it",
"[if it can only heat
one room] it will be a
problem", "No, I
would not buy it",
"This method takes
more time for us as
well as you [the
scientists]"
"Could you make a
stand for the top of
the stove?", "How
often reveals
needs or wants
regarding a
specific concept
wants as well as potential pitfalls in our
design. Many of these questions could
be interpreted into a specific need or a
form of criticism, and yet it was
long could you cook
for using the box?",
"What maintenance
does it require?",
somewhat easier to elicit these type of
questions than to elicit criticism.
"Can it work when
it's rainy?", "Can a
single charge last for
a month?"
"I can boil milk with
it", "We can reheat
food", "It would be
best if it could fry"
Ideas
User suggests an
idea about the
concept
This could be difficult to get from
ordinary users, although expert
interviewers often had a number of ideas.
Confusion
User expresses
confusion over a
Occasionally, users did not understand
concepts or did not feel informed enough
"We are old, we don't
know", "We are
concept, or feels
they cannot give
informed feedback
to give feedback about concepts, and
openly said so.
confused about it", "It
is not clear how this
would be"
55
Specific
Needs/Wants
User directly
specifies what
they need or want
out of a certain
product
Sometimes, users told us directly what
they expected out of a product.
"It is important that
this technology is
safer than LPG", "We
should be able to use
it directly, without
pipes"
Other
Other types of
comments, for
instance, user
provides
information or
"People spend a lot of
money on solar, but
then it breaks and
they become
discouraged", "Ifyou
opinions that they
make people really
think may be
relevant to the
project, or
speculates about
what others might
think of the
product.
understand how it
works, then maybe
people will buy it"
Table 4-1: Categorizing the different types of feedback
Useful misunderstandings:
Also worth noting are useful misunderstanding. Misunderstandings were expressed in any of the
above types of feedback, so it is not its own category. However, on occasion the types of
misunderstandings that occurred were very useful in inspiring the designers to think of ideas they
hadn't thought about before. For instance, during a group interview in Jaipur, both the translator
and the users misunderstood the storyboards and put their own unique interpretation on how the
solar thermal fuel could be used. They viewed the system as a box with replaceable "heat
batteries" whereby, rather than charging the entire device, they could remove the "batteries" and
charge just those.
4.2
Coding the Feedback
To process the feedback and extract patterns from it, we combined our qualitative fieldwork
experience with a more quantitative method of coding. We used a method of a combination of
pre-set and emergent coding to determine the categories of feedback. The author and a research
assistant individually went through the transcripts and notes and categorized the feedback, and
got fairly similar results. Then, we were able to use these results to help guide and verify our
qualitative observations.
56
4.3
Variables Affecting Feedback
Although there are many influencing factors at work, such as cultural background of the
interviewer, we identified three broad sets of factors that seemed to affect our studies the most:
1) Method (focus group, group interview, one on one interview, workshop)
2) Prototype used (for this study, discussed in terms of the axes of dimensionality and
narrative level)
3) Demographic of user (age, education, social class, area of India, urban/rural, gender). It
is worth noting that in almost every single study we did, particularly in rural areas, older
people were less educated, so it is difficult to separate the effect of age versus education
level. We did not always ask about education level, since it was not always appropriate,
but when we asked users to sign consent forms for the study, many of the older users
asked to use their thumbprint. So for our study, we chose to focus mainly on age and
whether users were urban or rural.
We used the coded results, combined with the qualitative analysis from the studies
themselves to identify trends we saw in the types of feedback relative to the three factors
described.
We eliminated the ideas category because although ideas from users can be very useful, after
coding we realized that it was only in the focus groups where users were actively encouraged
through an activity to come up with ideas. In the other studies, we merely asked if they had any
other ideas rather than putting emphasis on it.
To better understand how these factors affect the feedback, we have also categorized each
study below by the method used, the type of prototype used, and the age category of the
participants and whether they were urban or rural. Outside of the focus groups, most of the
people we interviewed were of similar social class, so we do not categorize by that. For the
group interviews in Ladakh, groups typically consisted of only 4-5 people.
Study
Method
Type of
Prototype
Age Group
Urban vs. Rural
Delhi #1
Delhi #2
Jaipur #1
Focus group
Focus group
Group interview
None
None
Storyboards
Young - middle
Young - middle
Young - middle
Urban
Urban
Rural
57
Jaipur #2
Group interview
Storyboards
Young - middle
Rural
Jaipur #3
Group interview
Young - middle
Urban
Jaipur #4
Group interview
Props +
Storyboards
Props +
Storyboards
Young - middle
Urban
Ladakh #1
Small Group
Storyboards
Middle
Urban
Ladakh #2
interview
Small Group
Props +
Young - Middle
Urban
interview
Storyboards
Individual
Storyboards
Old
Rural
Storyboards
Old
Rural
Storyboards
Young
Rural
Storyboards
Old
Rural
Storyboards
Middle
Rural
Storyboards
Old
Rural
Storyboards
Young
Semi-rural
Young, middle,
old
Young, middle,
old
Young, middle,
old
Rural
Ladakh #3
interview
Ladakh #4
(Monastery)
Small Group
interview
Ladakh #5
Individual
interview
Ladakh #6
Individual
interview
Ladakh #7
Individual
interview
Ladakh #8
Individual
interview
Ladakh #9
Individual
interview
Hubli #1
Workshop
Storyboards
Hubli #2
Workshop
Storyboards
Hubli #3
Workshop
Roleplay + Props
Rural
Rural
Table 4-2: Categorizing the different studies
4.3.1
Feedback Results
The results are categorized below in the following sets of tables, which look at the types
of feedback, and the factors that affected or led to that them. Note that the Needs/Wants
category is not included in the analysis, since we were unable to find any clear patterns for any
of the three factors we looked at.
58
Method Used
4.3.1.1
Feedback
Method
Preference
This type of feedback occurred naturally when presenting more than one
option; sometimes we asked for preferences (i.e. STF storyboard), but
sometimes we didn't and people expressed a particular choice anyway, such
Nonspecific
positive
as with the soil collection methods.
If we sort the number of non-specific positive comments in ascending order,
the individual interviews yielded more of this type of comment than group
interviews; however, we believe based on our experience that this had more to
do with the age group - most of the group interviews were conducted with an
averaged of middle aged people.
Specific
positive
The two focus groups gave the largest amount of specific positive feedback.
It is unclear if this was the result of demographics, the method, or any other
factor.
Criticism
People in group settings seemed much more willing to voice criticism.
Questions
We got far more questions in group settings, not only because the number of
people, but because one question tended to spark another.
Confusion
No clear effect.
Table 4-3: Analysis of feedback versus method
Not unsurprisingly, people asked more questions and give more criticism in group
settings. This matches with some of the literature on user feedback methods in the developing
world - for instance, Medhi, Sagar, and Toyama found in designing user interfaces for illiterate
users in India that when conducting studies in groups, women were more comfortable, bounced
ideas around more, and were generally louder and more confident in expressing their views [37].
In contrast, they found during single interviews, women were more nervous.
59
4.3.1.2 Prototype
Feedback
Type of Prototype
Preference
User understanding of the product here was very important, since users might
misinterpret a representation, or not fully understand all aspects of a concept.
For instance, in the case of the soil method collection concepts, it was only by
role-playing with physical mockups that we were able to properly convey how
laborious the different tasks were. When they did not understand the other
methods, they tended to go with the most familiar by default. Similarly,
people seemed to have an easier time making a preference for the storyboards
over the single panel images, because in some cases, they stated that they
didn't really know how the single panel images worked.
Nonspecific
positive
Specific
No clear effect.
No clear effect.
positive
Criticism
The user must have a clear enough vision of the product to foresee problems,
so the quality of the criticism was affected by how well the prototype
conveyed understanding.
Questions
People seemed to ask more questions when presented with a physical
prototype. However, this was not always the case - there were other group
settings were the physical prototypes were not used and we got asked many
questions. The types of questions we were asked when using the physical
Confusion
prototype generally differed.
Poor or inappropriate representation not unsurprisingly resulted in confusion.
There was more confusion when there were no prototypes (as in the focus
groups), or with the 2D images.
Table 4-4: Analysis of feedback versus prototype
4.3.1.3
Demographic (Age, Education, Rural/Urban)
Feedback
Demographics
Preference
We found in Ladakh, older/less education people were more likely to say they
liked everything, likely because they did not always understand the concepts.
The location was also important in that it often related directly to how familiar
users were with a similar technology. For instance, in Ladakh, users were
already very familiar with solar technology, and therefore found it easier to
give informed feedback about which product concept they preferred than users
I in Jaipur.
60
Nonspecific
positive
Older/less educated people tended to give non-specific positive feedback,
often without much meaning or understanding; it seemed often like they were
trying to please us.
Specific
positive
The two focus groups gave the largest amount of specific positive feedback. It
is unclear if this was the result of demographics, the method, or any other
factor.
Criticism
Younger/more educated people were more willing to voice criticism.
Questions
Younger/more educated people were more likely to ask questions
Confusion
Older/less educated people tended to get more confused, and sometimes
seemed embarrassed or nervous to offer their opinions. Additionally,
locations where users were less familiar with similar technology (as in Jaipur
with solar technology), users were more confused by product concepts. In
Jaipur, there was also a rural/urban divide, which may also have had to do
with education, but the urban users we spoke with were familiar with solar
1 technology.
Table 4-5: Analysis of feedback versus demographic
4.3.2
Limitations
The shortcoming of the coding method is that only some interviews were translated and
transcribed, while the rest were notes that were written on the spot during the interviews. In
addition to being a level removed from the original comments because of translation, this latter
set of notes feature both paraphrasing as well as exact quotes. Additionally, the different
interviews had different numbers of people, so the numbers are not directly comparable. Lastly,
because of the ambiguous nature of feedback, there were certain comments that did not
obviously fall into a single category. Because of this, there were some discrepancies between the
results of the two coders. Since the goal of the coding was not to get the exact numbers, but to
do a qualitative analysis of what types of feedback came out during the interviews, we found the
results to be adequate, and that the patterns observed still hold.
4.4
Qualitative Effect of Prototypes
Generally from our data, it is difficult to tell how much the type of prototype affected the
feedback versus the other factors. For instance, we got critical and insightful questions about the
solar thermal fuel from the group interviewed in Ladakh with the "magic box" as the group
interview in Jaipur with only the storyboards.
61
However, what we did find was that the type of prototype affected our level of
confidence in the user's feedback and the user's understanding of certain aspects of the product.
Revisiting our original hypotheses about prototypes, this is what we found.
4.4.1
People were able to extrapolate from prototype to images.
People fixated much less than expected with soil prototypes, and with magic box were able to
un-fixate with aid of storyboards after. Although in both these case studies, it seems as though
the 3D prototypes conveyed clearer understanding, we do not believe that using a 3D prototype
is always the best solution. Most people in Ladakh had no difficulty understanding the solar
thermal fuel storyboards, and so a physical prototype was generally unnecessary. As mentioned
in the section about the first case study, the one time we did get a chance to use the "magic box"
in Ladakh, we did find that people asked more discerning and targeted questions, and that
although they fixated on the idea of a cooker, they were easily able to "un-fixate" with the aid of
the storyboards after. In this case, it would have been inconvenient to present other physical
mockups such a roof-mounted system, which might have helped avoid fixation, but the
storyboards seemed to serve the same purpose.
In the second case study, there was much less fixation on details of the props than we
expected, perhaps because the low fidelity, or because we presented several options at once. In
contrast, the storyboards generated a lot of confusion.
Another interesting aspect of 2D versus 3D was that the types of questions asked differed.
Particularly with the "magic box", the questions were more similar to what someone might ask
before purchasing a product in terms of their specificity. For instance, the group in Ladakh
asked questions such as "How long could you cook for using the box?". In contrast, while using
the storyboards in Ladakh, the questions tended to focus on clarifying the concept, such as "Is it
like the solar water heater we already have?" Similarly in Hubli, questions asked when we used
physical prototypes were much more akin to the specificity of the questions asked when we used
the "magic box".
Interestingly, in Jaipur we found that regardless of the prototype used, some similar types of
questions came up, such as "Is the liquid harmful if split?" So, it is not clear how generalizable
our observation about the types of questions is, or what other factors may be in effect.
62
4.4.2
Storyboards helped users visualize the product more holistically in some cases, but
caused confusion in others.
In the case of the solar thermal fuel, where the product could come in any number of form
factors, the physical prototype caused to fixate people upon a single concept, whereas the
storyboards helped them visualize the technology more holistically. However, in the case of the
soil collection methods, the storyboard failed to convey a good enough understanding of the
method of collection to the degree that people were able to envisage other form factors.
4.4.3
Using the right prototype.
Different types of prototypes demonstrate different aspects of a concept. We learned that
prototype must ensure that the user has a firm understanding of the more critical aspects of the
concept. For instance, in demonstrating the different methods to collect soil, we showed
prototypes in the form of storyboards, and in the form of a role-play with simple props. The
role-play turned out to be far more effective, since one of the key differences is the labor
intensiveness of the different methods. Images alone were unable to convey this, but by actually
asking users to dig a hole, they had a much clearer understanding of the method. However, that
is not to say that roleplaying is always the best method.
A paper by Wiklund, Thurott and Dumas finds that, when paper prototyping software, it was
important to have the prototype mimic the response time of the design concept - "When the real
product produces slower response times, prototype performance may give an overly optimistic
picture of the product's usability." [38] Similarly, we found that conveying time and labor was
similarly important in representing soil collection methods. The storyboards failed to convey the
amount of time and labor taken for some of the methods, and it was only once role-played that
the users seemed to understand the drawbacks of some of the more labor and time intensive
methods. Although we did not get an opportunity to test it with the solar thermal fuel, the time
taken to charge and use the product concepts also play an important part in the usability;
however, given the users' familiarity with solar products, they seemed to have understood this
just from the storyboards.
4.4.4
Did the narrative aspect matter?
In cases where users were able to analogize easily to similar products, such as the STF heaters,
the narrative aspect seemed to matter less. However, in all other cases, the narrative aspect
63
seemed to help in conveying understanding. For instance, in every single interview with the
solar thermal fuel, when presented with the two storyboard choices, people always made a
choice, but when presented with the single panel images, several people said they were fine with
any of them, or said they didn't know enough to make a choice. This may have had to do with
the fact that the single panel images analogized from existing products, but we speculate that it
may have also been the lack of narrative to contextualize the product. However, overall it is
unclear to us to what degree narrative aspect of the prototype mattered, since the prototypes used
in case studies make it difficult to compare the effect of the prototypes used directly.
4.4.5
Confidence in feedback
What we found was that the type of prototype we used affected our confidence in the type of
feedback people gave. Based on the types of questions asked and other responses, it was
generally evident when the users did or did not have a clear understanding of the product
concept. For instance, in the case of soil collection methods, the storyboards made it difficult to
really understand how much physical labor would be involved, whereas roleplaying with simple
mockups made it very clear; it was clear to us that the feedback from the users was far more
informed as a result.
4.5
Other Aspects Affecting Feedback
4.5.1
Familiarity with Solar Technology
In Ladakh, all of the people we spoke to were at least somewhat familiar with solar technology
(solar hot water heating, and solar lighting in particular). This made it much easier to
communicate the idea of a new solar technology, partially because people understood solar
technology in general, and partially because we could use existing technology as an analogy.
We found these analogies to be helpful later on in Jaipur when we were able to communicate the
idea of a cooker that heats by conduction by comparing it to an induction cooker.
4.5.2
Micro-entrepreneurs
Austin-Breneman and Yang recommend targeting developing world products towards microentrepreneurs [20]. In keep with this, we spoke to many hotel and guesthouse owners in Leh and
nearby villages, since tourism is a large industry in Ladakh. The owners of the hotels that we
spoke to seemed especially knowledgeable about solar hot water heating, and their current
64
expenditures on water and room heating. In addition to hotel owners, we also spoke to vendors
of solar products such as solar hot water heating and solar lighting. They gave us a good idea of
how and why new solar technology is being adopted, and recommendations on what we might do
with the STF.
4.5.3
Gender
Although we did not have a chance to compare in as much depth, there was a noticeable
difference between how men and women responded. In Jaipur especially, we noted that
generally men dominated the conversations in group settings. Even when interviewing only
women, the women sometimes deferred to the translator to answer for her. This disparity was
less pronounced during the Hubli workshops, but still there.
4.5.4
Cultural Background of the Interviewer
As mentioned in Section 2, Vatrapu and Pdrez-Quifnones found that the cultural background of
the interviewer affected the feedback [17]. It is likely, then, that in our studies, that this had a
similar effect. What was not studied in the paper was the effect of having a local translator with
a non-local interviewer, so we do not know if even our presence changed people's willingness to
respond openly.
4.5.5
Translation
Translation turned out to be a major difficulty throughout the entire process to the degree
of affecting the feedback we documented. We often did not have professional translators since
we were reliant on local contacts to arrange logistics. There were several difficulties that
resulted: difficulties with English, biased translations, misunderstanding of the technology, and
answering the questions for the interviewee. For instance, one translator had somewhat poor
English, as so often misunderstood our interview questions as well as the technology we were
trying to discuss. He also had a tendency to supply his own opinion rather than letting the
interviewee answer, and often seemed like he was aiming to please us, so we do not know if his
translations were overly positive.
Even with more reliable translators, some of these difficulties still occurred. For
instance, our translator in Ladakh, although not professional, was an engineering student with
excellent English. We had almost no difficulties with his translations. However, at one point, it
65
became apparent that he had misunderstood one of the technologies described in the storyboard,
and was explaining it incorrectly.
During the second case study, we had other difficulties with translation. Although the
translators were on the whole excellent, there were still a few difficulties. The main two
problems with translation during that study was that the translation often broke the flow of
conversation, and the translations we got were generally very short as compared to what was
actually said.
66
5 General Discussion
5.1
Observed differences
We discussed in Section 2 what some of the differences between emerging market users and
conventional users are. We also had the opportunity to build upon the existing literature with out
own observations, specific to India.
1) Education. Many of the older users we spoke with in rural areas had no formal
education and were often not literate. We found generally that educated users tended to
be more capable of comprehending more hypothetical concepts, and were thus able to
better visualize product concepts better. This matches the feedback results we saw.
2) Exposure to technology. As was discussed previously in Section 2, it was not always
the case in the developing world that users have been exposed to advanced technology;
although cellular phones are becoming increasingly ubiquitous, we visited communities
where the arrival of electricity was recent, and still unreliable, and where cooking with
cow dung in a clay hearth was the norm. Similar to Maunder and Marsden's experience,
we found it can be far more difficult to ask a user to envision a fully functional high-tech
product and how it might fit into their lives than it would be in the US [9]. This isn't
always the case though - in one of the locations we visited, even illiterate rural farmers
were extraordinarily familiar with solar technology, thanks to the efforts of the local
government. During one interview, a neighbor passing by interjected to say she'd seen a
demonstration of a trombe wall in a nearby village - whereas the author of this paper
hadn't even heard of a trombe wall until embarking on this research!
3) Infrastructure. Infrastructure in the developing world poses another barrier. It varies
from country to country and region to region. For instance, in India, under-developed
roads in India make supply chains difficult, and lack of electricity in rural areas can make
products that use electricity pointless. However, for the urban middle class, these may
not be barriers. Urban slums pose a whole different set of challenges as well.
4) Expectations of charity. One thing we were warned about by a local NGO employee
was that in Ladakh, so many charities and NGO's went through that locals often
anticipated a free or subsidized product, and might therefore be inclined to say they like a
product regardless [39]. Similarly, a student at IITB told us that in her experience
67
interviewing people in rural areas, people would occasionally exaggerate to make
themselves appear in worse circumstances, with the hopes of being the recipient of some
donation [40]. She had an experience where she asked a woman about the size of the
farm and the woman responded with one number; however, her son was passing by, and
(to the chagrin of his mother) said that their farm was actually larger. Therefore, this
problem isn't limited only to visiting foreigners! Our translators told is on several
occasions that they believed interviewees were being nice because they thought we'd
give them a free product.
5) Cultural differences. The cultural differences between the US and India is an entire
topic on its own, but suffice to say, there were many! One thing of particular note was
the deference some Indians displayed to us as foreigners, which likely affected feedback.
The combination of lack of education and lack of exposure to technology makes conveying
product concepts particularly difficult. Even before conducting studies and observing the
differences in users for ourselves, we were unsure how well users would respond to low-fidelity
prototypes. Would users understand that the prototypes were not meant to be the final product,
but only meant to demonstrate certain facets of a concept? Would they be able to extrapolate
from a simple prototype what a fully functional product might look and work like, and how it
would fit into their lives? Even if they didn't understand the underlying technology, would they
have a good enough sense of how a product could work to provide useful feedback?
5.2
5.2.1
Other Takeaways
Intro
In addition to the results and discussion presented earlier, there were a number of other areas of
interest and practical issues that came up that are worth discussing.
5.2.2
Getting realistic responses
One thing we were warned about early on from Greg Tao was getting realistic responses [15]. In
general, there is a tendency to be overly positive and an unwillingness to criticize [6]. Although
It cai be IIIcUltL Lo kno
fLr aLUaL uesiU1ns Ik tese how honest he
i4terviewee is being,
for questions about opinions (i.e., what do you think of this product?), there are a number of
clues:
68
1) Understanding. There was more than one interview where it became clear that the
interviewees did not have a good understanding of the concepts we were discussing.
However, they would still tell us that they thought it was a great idea and they'd buy it.
2) Translator's intuition. With some of the more reliable translators we had, they were able
to tell us if it seemed like the interviewee was giving a thoughtful opinion, or if they were
just trying to please us.
3) Vague answers. In several cases, users would ask a number of very targeted questions
showing them to be thoughtful and discerning consumers, but then would finish the
conversation by expressing vaguely that everything would be useful.
4) The interviewer. As mentioned earlier, having the interviewee be someone from the
same cultural background can change how freely people respond to the interviewer [17]
5.2.3
Difficulty with abstract ideas.
There were two different types of abstraction we dealt both. The first is abstraction of the
product, which is what our research primarily focused on. In both cases, we did not have a
working prototype of the product to demonstrate. Instead, we had to use abstractions such as the
storyboards, images, 3D props (magic box, foam models), and 2D props (the soil test cards).
In the case of the first type of abstraction, while several people expressed annoyance at
lack of the real product, or felt they could not make a real judgment call without seeing the real
thing, it was possible to convey these concepts.
The other form of abstraction was the abstraction of concepts. This came up when we
were trying to better understand the product attributes (such as cost, simplicity, durability) that
mattered most to users. People in rural areas had far more difficulty with this type of abstraction,
as evidenced by what happened with the product attribute exercise.
5.2.4
Making sure to interview a variety of people - users and other stakeholders; get a
very different story
This is generally a good rule of thumb in doing user studies [41], but we found in India it was
especially important to interview both experts and the users, since we occasionally got vastly
different stories. The most notable was in talking to the people in the government office who do
soil testing. According to them, they processed a large number of tests regularly. However, all
69
of the farmers we spoke to who had sent their soil for testing said they never got the results back
(and in some cases, it had been years since they'd sent their samples!).
5.2.5
Observations combine with interviews
Doing a combination of observations and interviews was a good idea. In Ladakh, by staying
overnight with farmers, we got a much better impression of their lifestyle, which helped guide
interviews questions later. In Hubli, we didn't have any time for observations and jumped
straight into interviews; this made it a bit more difficult to really understand the experience of the
farmers, and also meant we didn't have much in the way of observations to validate what people
were saying.
5.2.6
Giving multiple choices to choose from
It can be difficult to get criticism from users. When we were trying to get feedback about the
solar thermal fuel, we had several concepts we wanted to test. What we found was that by
asking about several concepts rather than just one, even if people were unwilling to tell us if they
didn't like any option, they still gave us a sense of relative preference. This matches what saw
during their usability tests as well [25].
5.2.7
Village Politics
One lesson we learned during the workshop was to be wary of how village politics may influence
interactions. For two of the three workshops, most people were from different villages and did
not know each other. However, the second workshop, most of the participants came from the
same village. As a result, they knew how much land other people owned, how much wealth,
how educated they were, etc. - all aspects that influenced their interactions and may have
contributed to the overall feeling of the second group being more "negative". Smith and Leith
also warn about how local politics may affect user interactions in their handbook [18].
5.2.8
Looking out for misunderstandings
The possibilities for misunderstandings abounded, both between the designers and translators,
and the participants and translators. Sometimes we were able to catch the misunderstandings
because the feedback, but other times we only caught them after asking the translator to repeat
what they told the interviewee. There were very likely other misunderstandings we didn't catch
at all.
70
5.2.9
Attitude
Go in with the attitude that you are there to learn from the people you are working with. You
may be the expert about engineering, but they are the expert about what they need, and by
working with them rather than trying to solve their problemsfor them, you will have a more
realistic idea of what people actually want.
5.2.10 Translation
We already discussed some of the difficulties we had with translation. We propose a few
suggestions to avoid some of these difficulties, particularly in group or workshop settings.
1) Have a second translator who just takes notes, rather than the translator trying to translate
for the note taker.
2) Training the translators to moderate or otherwise take more responsibility for the flow of
the conversation
3) Have someone on the project who can speak the local language. People will be more
forthcoming with a local in any case [17].
4) Work in smaller groups, or design activities that don't depend highly on a good translator
5) Rehearse the workshop with the translators ahead of time. (This really should be more
than a suggestion, but a mandatory guideline!)
5.2.11 Good local contacts.
Having a strong local contact was perhaps the most important part of the entire experience. In
the week we spent in Hubli, we got more done than we did during the 4 weeks in Ladakh. This
was largely due to having a strong local contact that was able to arrange interviews and
workshops with locals and provide us with translators.
5.2.12 You are a tourist attraction
In certain areas we visited, foreigners were enough of a rarity that brought half the village
(especially children!) to be spectators during our interviews. This made it very difficult at times
to hold conversations, especially in rural areas where the interviews were often held outdoors.
Parikh, Ghosh, and Chavan describe a similar experience in their paper, and found that by
staying in the village they were working in for a longer period, the novelty gradually wore off
and they were able to have more normal interactions [24].
71
5.2.13 Adaptation
One lessons we learned was that nothing ever goes according to plan! We were able to adapt on
the fly but one thing that really helped with this was planning different approaches ahead of time.
Since we did not know what to expect of the workshop, we had several versions of each activity
planned.
5.2.14 Being on the ground
The most important thing is to actually be on the ground and interacting with the people you are
designing for! Otherwise, you will not get a good understanding of your target market. Similar
to Parikh, Ghosh, and Chavan's experience, we found that we needed to be on the ground to get
a good understanding of the culture and lifestyle [24]. They also point out that in their case, by
remaining in the village for two months, villagers got to know them well enough that they felt
comfortable in giving more critical feedback.
72
6
6.1
Summary
Conclusions
If we look back at our two original research questions, this is what we find.
1) How does the method use affect the quality and type of qualitative data gathered?
For the methods used, the type feedback elicited generally followed what might be expected
in conventional settings, although the quantity and quality of the feedback likely differed. In
particular, methods involving group settings resulted in people being more comfortable
voicing their thoughts. However, methods that involved a great deal of abstraction
sometimes confused users.
2) How does the demographic of the user (age, education, social class, area of India,
urban/rural, gender, etc.) affect feedback?
In general, users who were more educated were less likely to give overly positive feedback,
more likely to ask questions and voice criticism. In many of our studies, older people in rural
areas tended to have little to no formal education, so this usually meant younger and middle
aged users were the most valuable source of feedback. The location of India as well as
whether users were from rural or urban locations often affected their exposure to certain
technology, which also affected feedback.
3) How does the type of prototype used affect user feedback?
We were surprised to find that the barriers to discussing abstract technological concepts were
lower than we expected, and even with our low fidelity prototypes, we were still able to
collect a great deal of data. Users in emerging markets were generally able to understand
that the prototype was only meant to represent a product. People did not fixate on details of
the physical prototype as much as expected, but it did cause some people to view the physical
prototype as the only form factor possible, until shown storyboards. However, storyboards
73
had a higher potential for causing confusion, especially in cases where users were not
familiar with any similar products. Ultimately, there is no single "best" type of prototype.
The type of prototype needs to be appropriate to the concept being tested.
6.2
Areas that need to be further researched
Throughout the course of writing this thesis, more questions than answers came up. A few of
these areas that need to be further explored are described below.
6.2.1
Controlled experiments with prototype types
Because this project focused on real world case studies, it was difficult to do more controlled
experiments with the different types of prototypes we used, since the focus was on usability
testing for the product rather than on testing methods. It would be interested to see more
rigorous experiments where the main focus was on studying how the type of prototype affects
the feedback.
6.2.2
Moving towards participatory design with more complex technology
Some of the literature for design for development encourages co-design and participatory design
as a way to improve the appropriateness of the design as well as empowering the users [13][14].
The workshop we did in Hubli was not participatory design so much as usability testing. In
some ways, our mentality is similar, in the assumption that the users have just as much to
contribute to the project than the designers, since they are experts on their needs. However, for
this project we did not ask them to actively participate in the design process; rather, we focused
on trying to encourage a conversation around feedback on the early stage concepts developed by
the designers, early stage needs assessment, and trying to understand what attributes of a product
matter to the users. The implicit understanding was that as designers, we are experts on the
technology and engineering, while the users are experts on the needs for soil testing and what
they expect of a product.
The reason we choose to focus on usability testing over more open participatory design
largely due to the nature of the project. Given that the technology drove the project to some
degree, it made sense to evaluate early concepts for how the technology would work rather than
askn111g uls1ers t'
generate ojLte-
pts. Additionauly, t11he techlogy itself
s difficult to
understand without expertise in that area, so to ask users to try to participate in designing the
product itself seemed unreasonable. Future research could focus on the question of how
74
designers can go about engaging users in the design process when there is complex technology
involved.
6.3
Ramifications
With a growing number of university courses aimed at product development for global emerging
markets, we hope our work can inform user research studies and thereby improve the overall
quality of products for emerging markets.
75
7
Bibliography
"PopTech, Kevin Starr of the Mulago Foundation. 2011 [online]. Available:
http://poptech.org/blog/kevin-starr-demands-impact. [Accessed: 10-April-2014]."
[2]
S. Boettiger, T. Straus, A. Conner, K. Carter, K. Pope, and H. Fleming, "Demand-Driven Innovation," 2015.
[3]
A. L. Chavan, D. Gorney, B. Prabhu, and S. Arora, "The Washing Machine That Ate My Sari in Cross-Cultural Design," Interactions,no. January + February, pp. 26-31, 2009.
[4]
D. A. Norman, The design of everyday things /DonaldA. Norman. Cambridge, Massachusetts: MIT Press,
.
[1]
Mistakes
2013.
[5]
C. Oyugi, L. Dunckley, and A. Smith, "Evaluation methods and cultural differences," in Proceedingsof the
5th Nordic conference on Human-computer interaction building bridges - NordiCHI '08, 2008, p. 318.
[6]
C. Courage and K. Baxter, UnderstandingYour Users: A PracticalGuide to User Requirements Methods,
Tools, and Techniques. Morgan Kaufmann Publishers Inc., 2004.
[7]
M. McCurdy, C. Connors, G. Pyrzak, B. Kanefsky, and A. Vera, "Breaking the fidelity barrier," Proc.
SIGCHI Conf Hum. FactorsComput. Syst. - CHI '06, p. 1233, 2006.
[8]
S. Houde and C. Hill, "What do Prototypes Prototype?," Handb. Hum. Comput. Interact., pp. 1-16, 1997.
[9]
A. Maunder, G. Marsden, D. Gruijters, and E. Blake, "Designing Interactive Systems for the Developing
World - Reflections on User- Centred Design," in 2007 InternationalConference on Information and
Communication Technologies andDevelopment, 2007.
[10]
G. Tao, "Design of a Low-cost Autoclave for Adoption in Rural Health Posts of the Developing World,"
MS thesis, Dept. Mech. Eng., MIT, Cambridge, MA, 2010.
[11]
A. Yeo, "Cultural Effects in Usability Assessment," in Proceedingsof the SIGCHI conference on Human
factors in computing systems, 1998, no. April, pp. 74-75.
[12]
K. M. Donaldson, "Product design in less industrialized economies: constraints and opportunities in Kenya,"
Res. Eng. Des., vol. 17, no. 3, pp. 135-155, Oct. 2006.
[13]
C. A. Mattson and A. E. Wood, "Nine Principles for Design for the Developing World as Derived From the
Engineering Literature," J. Mech. Des., vol. 136, no. 12, p. 121403, Oct. 2014.
[14]
D. Nieusma and D. Riley, Designs on development: engineering, globalization, and socialjustice, vol. 2, no.
1.2010.
"Interview with Gregory Tao, Co-founder of Ottoclave, March 6 , 2014."
[16]
T. Vainio. T. Walsh, and I Varsaluoma, "Cross-Cultural Design of Mobile Mathematics Learning Service
.
[15]
for South African Schools," IADIS Int.
J. WWW/Internet, vol. 12, no. 1, pp. 81-93, 2014.
76
R. Vatrapu, M. A. Pdrez-Quifiones, and V. Tech, "Culture and International Usability Testing: The Effects
of Culture in Structured Interviews," J. Usability Stud., vol. 1, no. 4, pp. 156-170, 2006.
[18]
R. Smith and K. Leith, "D-Lab Scale-Ups User Research Framework." Cambridge, p. 53, 2014.
[19]
J. Nielsen, "International Usability Testing," 1996. [Online]. Available:
http://www.nngroup.com/articles/intemational-usability-testing/.
[20]
J. Austin-Breneman and M. Yang, "Design for Micro-enterprise: An Approach to Product Design for
Emerging Markets," in Proceedings of the ASME 2013 InternationalDesign EngineeringTechnical
Conferences & Computers and Information in EngineeringConference, 2013, p. 11.
[21]
M. W. Tschudy, E. A. Dykstra-Erickson, and M. S. Holloway, "PictureCARD: A Storytelling Tool for Task
Analysis," in Proceedingsof the ParticipatoryDesign Conference, 1996, pp. 183-191.
[22]
R. Sefelin, M. Tscheligi, and V. Giller, "Paper Prototyping - What is it good for? A Comparison of Paperand Computer-based Low-fidelity Prototyping," CHI '03 Ext. Abstr. Hum. Factors Comput. Syst., pp. 778779, 2003.
[23]
R. a. Virzi, J. L. Sokolov, and D. Karis, "Usability problem identification using both low- and high-fidelity
prototypes," Proc. SIGCHI Conf Hum. factors Comput. Syst. common Gr. - CHI '96, pp. 236-243, 1996.
[24]
T. Parikh, K. Ghosh, and A. Chavan, "Design studies for a financial management system for micro-credit
groups in rural india," Proc. 2003 Conf Univers. usability - CUU '03, p. 15, 2003.
[25]
M. Tohidi, W. Buxton, R. Baecker, and A. Sellen, "Getting the Right Design and the Design Right: Testing
Many Is Better Than One," in ProceedingsCHI 2006 Conference: Human Factorsin Computing Systems,
2006, pp. 1243-1252.
[26]
B. Macomber and M. Yang, "The Role of Sketch Finish and Style in User Responses to Early Stage Design
Concepts," in Volume 9: 23rd InternationalConference on Design Theory and Methodology; 16th Design
for Manufacturingand the Life Cycle Conference, 2011, pp. 567-576.
[27]
R. K. Yin, Case study research:design and methods. Los Angeles: SAGE, [2014], 2014.
[28]
"Personal Communications with Subarna Basnet of MIT, April, May 2014."
[29]
E. Rehfuess, S. Mehta, and A. Prtiss-Ost~n, "Assessing household solid fuel use: Multiple implications for
the Millennium Development Goals," Environ. Health Perspect., vol. 114, no. 3, pp. 373-378, 2006.
[30]
R. Mukhopadhyay, S. Sambandam, A. Pillarisetti, D. Jack, K. Mukhopadhyay, K. Balakrishnan, M.
Vaswani, M. N. Bates, P. L. Kinney, N. Arora, and K. R. Smith, "Cooking practices, air quality, and the
acceptability of advanced cookstoves in Haryana, India: an exploratory study to inform large-scale
interventions," Glob Heal. Action, vol. 1, no. July, pp. 1-14, 2012.
[31]
A. Doyle, "Air pollution scourge underestimated, green energy can help: U.N.," Reuters, Oslo, 2013.
[32]
D. A. Norman and R. Verganti, "Incremental and Radical Innovation: Design Research versus Technology
and Meaning Change," Des. Issues, pp. 1-19, 2012.
[33]
R. Gergan, "Development of Mini-Grids in the Ladakh Region of Jammu and Kashmir," Leh, Ladakh, 2013.
.
[17]
77
K. N. Truong, G. R. Hayes, and G. D. Abowd, "Storyboarding: An Empirical Determination of Best
Practices and Effective Guidelines," in DIS '06 Proceedingsof the 6th conference on DesigningInteractive
systems, ACM, 2006, pp. 12-21.
[35]
"Interview with Isaac Gergan of Ladakh Arts and Media Organization, July 2, 2014."
[36]
V. Goel, Sketches of thought. Cambridge, Mass.: MIT Press, c1995., 1995.
[37]
S. Users, I. Medhi, A. Sagar, and K. Toyama, "Text-Free User Interfaces for Illiterate," in International
Conference on Information and Communication Technology and Development, 2006, pp. 72-82.
[38]
M. E. Wiklund, C. Thurrott, and J. S. Dumas, "Does the fidelity of software prototypes affect the perception
of usability?," in Proceedingsof the Human FactorsSociety, 36th Annual Meeting, 1992, pp. 399-403.
[39]
"Interview with Ruchi Mathur of GERES in Leh, July 2, 2014."
[40]
"Personal Communication with Akanksha Nagpal of IIT Bombay, January 2014."
[41]
K. T. Ulrich and S. D. Eppinger, Product Design and Development, vol. 384. 2011.
.
.
[34]
78