Needs Analysis:
Design through Discovery
Philip Kortum, Ph.D.
Robert Bushey, Ph.D.
SBC Laboratories
Human Factors
SBC Laboratories
2/12/04
Slide 1
Today’s Agenda






Who we are (5 min)
Basic needs analysis techniques (60 min)
Virtual Lab tour (10 Min)
Break (10 min)
Needs analysis in high-data environments (70 min)
Conclusions (10 min)
SBC Laboratories
2/12/04
Slide 2
Who is SBC?
Short Answer:
•We’re the phone company
Long Answer:
•2nd largest telecommunications providers the US
•Wireless
•Long Distance
•Data
•Residential/Business phone service
•Over 60 million access lines
•$41 Billion in Revenue (# 27 on the Fortune 500)
SBC Laboratories
2/12/04
Slide 3
Who is SBC Laboratories?
SBC Laboratories is the Applied Research and Development arm of
SBC.
About 200 Advanced-degreed Scientists and Engineers work at SBC’s
laboratories in Austin and the Bay area in California.
Technology Focus Areas
Broadband
Architecture,
Infrastructure,
& Services
SBC Laboratories
Intelligent
Networks
Internet
2/12/04
Wireless
Systems
Information
Technology
Slide 4
SBC Laboratories
Human Factors Engineering
Stages in the Product and System Development/Acquisition Cycle
Detailed
Development
Human Factors Concept High-Level
Generation
Design/RFI
Design/RFP
and Testing
Activities
Release
Evaluation
User Needs
Assessment
User
Profile
Competitive
Analysis
System/User
Modeling
Usability
Specification
User Interface
Design
Usability
Evaluation
User Interface
Design Guidelines
Product
Integration
SBC Laboratories
2/12/04
Slide 5
Where today’s talk fits in this class
From Mayhew, 1999
SBC Laboratories
2/12/04
Slide 6
What is Needs Analysis?
Needs analysis is the method of
uncovering user requirements
through direct or indirect
interaction with that user
SBC Laboratories
2/12/04
Slide 7
Why do we perform needs analysis?


We are not the user
Users may not know what they really need
» User is unable to articulate the need
» True needs are often masked

Good match between needs and
implementation leads to superior
efficiency and usability
SBC Laboratories
2/12/04
Slide 8
How is Needs Analysis different than
Task Analysis?

Needs Analysis – trying to uncover the
underlying motivations

Task Analysis – trying to uncover the
procedural steps
SBC Laboratories
2/12/04
Slide 9
How Is Needs Analysis Different Than
Task Analysis?

Task Analysis
» Setting up a VCR
» Programming a VCR

Needs Analysis
» The ways a person uses a VCR and
the reasons behind that use
– What they record
– What they do with what they
record
– Time-shifting
– Remote control use use
SBC Laboratories
2/12/04
Slide 10
How Is Needs Analysis Different Than
Marketing?




They both deal with the user
They both try to determine what the user
‘needs’
Some of the techniques are similar
Some of the information collected is the same
SBC Laboratories
2/12/04
Slide 11
How Is Needs Analysis Different Than
Marketing?
Marketing

Demographics
» Where people live, shop
» How much they make
» Physical attributes

Purchasing behavior
patterns
Needs Analysis

User behavior patterns
» What people do



What users know
Users environment
Users mental models
» What people will buy
» Price points
SBC Laboratories
2/12/04
Slide 12
Needs vs. Marketing Example
A device that records off of the television
Marketing




How much money do the
users have to spend?
What other similar devices do
they have?
What kinds of technology do
they own?
How much do they use these
technologies
SBC Laboratories
Needs Analysis




2/12/04
Why would they want to use
an image recorder?
How do they use current
devices?
When do they use such
devices?
What other technology
interactions are there?
Slide 13
A Simple (if unfair) Characterization:
Marketing deals with wants
Needs deals with needs
SBC Laboratories
2/12/04
Slide 14
A word about Semantics….

Many techniques are similar and are
called different things by those who use
them (even within the usability
community)

Good practitioners almost always use a
mix of techniques, and often call these
mixtures by a new name.
SBC Laboratories
2/12/04
Slide 15
Needs Analysis
Can be very focused
and limited in scope
Doesn’t have to ‘big sky’
SBC Laboratories
2/12/04
Slide 16
Basic Types of Needs Analysis
Questionnaires
 Interviews
 Observational Research
 Hybrid methodologies

SBC Laboratories
2/12/04
Slide 17
Questionnaires
“A set of questions for obtaining
statistically useful or personal
information from users”
Webster
SBC Laboratories
2/12/04
Slide 18
Questionnaire Advantages

Easy to administer
» Inexpensive
» Can be administered by untrained staff





Can be administered remotely
Can have a relatively large number of
questions and data samples
Can more easily target highly specific
users
Can branch to reliably capture pertinent
information
Data can be simple to code
SBC Laboratories
2/12/04
Slide 19
Questionnaire Disadvantages


Difficult to construct a good test
instrument
Must know a priori exactly what you want
to ask
» No follow-up questions

Question intent may be open to
interpretation
» Can result in unknown bad results



Open ended responses highly variable
Open ended questions are difficult to code
Validation/verification difficult if study is
not closely controlled
SBC Laboratories
2/12/04
Slide 20
Questionnaire Disadvantages: Fixes

Difficult to construct a good test
instrument
» Follow scientifically designed questionnaire guides
(Kirakowsi, Gillham, USARI)
» Test for validity, reliability, repeatability

Must know a priori exactly what you want
to ask
» Conducted pilot tests to try out the questions
» Conduct interviews to help build a reasonable first pass
» Use domain experts to help capture detail questions
SBC Laboratories
2/12/04
Slide 21
Questionnaire Disadvantages: Fixes

Question intent may be open to
interpretation
» Conduct pilot tests to check for weaknesses
» Use experts and novices to examine interpretation
differences
» Use unambiguous, quantifiable, anchored language
SBC Laboratories
2/12/04
Slide 22
Ambiguous Question Example
How bad was your last car accident?
Really bad
bad
not really bad
not bad at all
2 respondents, who both dented the
front bumper slightly, no injury
Respondent 1: ‘Really Bad’
Respondent 2: ‘Not bad at all’
“I’m only 16, it’s my first accident
and it was my dad’s new sports car,
and I wasn’t on the insurance for the
car. I’m going to be grounded for
life!
“Well, compared to that 23 car roll-
SBC Laboratories
over collision I had with that semi
last month, this was nothing. Not to
mention that I got the collision
waiver on the rental!
2/12/04
Slide 23
Class: How could we fix this?
Questionnaire Disadvantages: Fixes

Open ended responses highly variable
» Provide hints, examples to guide the responses
» Use multi-part questions that are more precise

Open ended questions are difficult to code
» Use more choices, or more questions, to remove need
for open ended questions
» Focus on variables of interest and use ‘other’ to capture
low probability events that are not pertinent
» Devise keyword codebooks to help code open ended
data
SBC Laboratories
2/12/04
Slide 24
Questionnaire Disadvantages: Fixes

Validation/verification difficult if study is
not closely controlled
» Key code questionnaires to match user and instrument
» Lock-out multiple take attempts
» Ask qualifying questions as part of a pretest, or branch
SBC Laboratories
2/12/04
Slide 25
Interviews
“A meeting at which information
is obtained from a person”
Webster
SBC Laboratories
2/12/04
Slide 26
Interviews
Generally classified into two types:
•Structured
•Unstructured
SBC Laboratories
2/12/04
Slide 27
Interview Advantages





Small numbers of interviews can be
performed quickly and inexpensively
(although not always)
Can be conducted remotely
Can be conducted in groups or
individually
Can be conducted with little notice, if
needed
Can change the interview for different
levels of users
SBC Laboratories
2/12/04
Slide 28
Interview Advantages




Open for opportunistic data discovery
Get non-verbal clues to help guide the
interview
Good technique for ‘fishing expeditions’
Useful for gathering preliminary
information to guide later needs analysis
SBC Laboratories
2/12/04
Slide 29
Interview Disadvantages

Hard to conduct a ‘good’ interview
» Easy to lead the respondent, with both verbal and nonverbal cues
» Easy to unknowingly ask loaded questions
– “How does it feel to be ugly?”

Harder still to conduct a good group
interview
» herd mentality, strong leader

Hard to stay ‘up’ and focused for a large
number of interviews
SBC Laboratories
2/12/04
Slide 30
Interview Disadvantages




Data can be difficult to code and quantify
Can be large differences between
interviewers
Expensive and time consuming to
conduct large numbers of interviews
Usually a low upper limit to the number of
questions
SBC Laboratories
2/12/04
Slide 31
Interview Disadvantages
Interview Example
SBC Laboratories
2/12/04
Slide 32
Interview Disadvantages
Question
What was said
What the data suggests
How many offers
do reps make per
call?
Reps make 1 offer
per call
Almost no reps make
1 offer, some do 0
some do 2+
What kind of rep is
the least desirable?
‘order takers’ are the
least desirable
Order takers are good they drive accessibility
and are among top $
performers
What makes the
customer happy?
Taking lots of calls is
what makes
customers happy
No supporting data

Interview data may not match reality
SBC Laboratories
2/12/04
Slide 33
Interview Disadvantages: Fixes

Hard to conduct a ‘good’ interviews
» Use scientifically developed methods (i.e. Weiss’s
Learning from Strangers)
» Pre-develop a core set of questions
» Manage group dynamics
» Use hypothetical questions to elicit more detailed
answers

Hard to stay ‘up’ and focused for a large
number of interviews
» Limit the number of interviews
» Use tag teams, alternate sessions
» Use well trained ‘people persons’
SBC Laboratories
2/12/04
Slide 34
Interview Disadvantages: Fixes

Data can be difficult to code and quantify
» use core set of questions
» use data code books to help quantify responses
» use affinity diagrams to aid in coding

Can be large differences between
interviewers
» use single interviewer
» use visible or hidden teams
» conduct pilot training to level the interviewers
SBC Laboratories
2/12/04
Slide 35
Interview Disadvantages: Fixes


Expensive and time consuming to conduct
large numbers of interviews
Usually a low upper limit to the number of
questions
»
»
»
»

target exact population
supplement with other techniques
use remote techniques
use core question set, based on pre-test
Interview data may not match reality
» Always verify data
SBC Laboratories
2/12/04
Slide 36
Observational Research
“An act of recognizing and
noting a fact or occurrence…”
Webster
SBC Laboratories
2/12/04
Slide 37
Observational Research

Observation doesn’t have to occur in real
time

Observation doesn’t have to be visually
based
» audio recordings
» diaries
» telemetry (GPS, web logs, biometrics, equipment monitors)
(e.g. car
snooper)
SBC Laboratories
2/12/04
Slide 38
Observational Advantages







Good for understanding complex needs
Results are not defined by the design of the
method
Capable of capturing unknown or undocumented
behaviors
Gather data on specific artifacts
Can uncover inter-dependencies
If documented appropriately, can review
observations as many times as necessary
Can parse expert/novice distinction
SBC Laboratories
2/12/04
Slide 39
Observational Disadvantages







Some phenomenon is too infrequent to
catch
Heisenberg Principle (i.e. driving tester)
Can be more difficult to obtain a
reviewable record
Desired behavior may be interspersed
with other unrelated activity
Slow, expensive to collect data
Can easily get ‘off-target’
Data set can be overwhelming
SBC Laboratories
2/12/04
Slide 40
Observational Disadvantages: Fixes

Some phenomena are too infrequent to
catch
»
»
»
»

use high sample rates to find low probability events
‘Create’ low probability (disaster training)
sample in areas known to have these events
be lucky
Heisenberg Principle (i.e. driving tester)
»
»
»
»
»
observe for extended periods of time
use remote measuring techniques
use low/no interaction models
clearly understand the cost-benefit of direct interaction
work in the users environment
SBC Laboratories
2/12/04
Slide 41
Observational Disadvantages: Fixes

Can be more difficult to obtain a reviewable
record
» use alternate recording technologies
» work in teams
» Use advanced behavioral software (e.g. Noldus Observer)

Desired behavior may be interspersed with other
unrelated activity
» use specific artifact techniques
» use remote data collection techniques to time compress
» don’t assume that unrelated activity is actually unrelated (e.g.
using the restroom while waiting)
SBC Laboratories
2/12/04
Slide 42
Observational Disadvantages: Fixes

Slow, expensive to collect data
» use remote or automated collection where possible
» understand required sample size
» carefully select person/situation/setting to be observed

Can easily get ‘off-target’
» use pilot observations to help identify key behaviors
» be open to the fact that off-target behaviors may be
pertinent
SBC Laboratories
2/12/04
Slide 43
Observational Disadvantages: Fixes

Data set can be overwhelming
» use data reduction/consolidation techniques
– training samples
– affinity diagrams
– physical flow/sequence structures
SBC Laboratories
2/12/04
Slide 44
Observational Research Example
Watching Television
SBC Laboratories
2/12/04
Slide 45
Observational Research Example
Watching Television
Nathen, et al, 1985
SBC Laboratories
2/12/04
Slide 46
Hybrid Methodologies




Ethnography
Contextual Inquiry
Empathic Design
Participatory learning
– actors
– rotational management assignments
SBC Laboratories
2/12/04
Slide 47
What happens when needs isn’t done?
SBC Laboratories
2/12/04
Slide 48
What happens when needs isn’t done?
SBC Laboratories
2/12/04
Slide 49
Questions?
SBC Laboratories
2/12/04
Slide 50