Primary Data and
Measurement
Jeremy Kees, Ph.D.
Stages
in the
Research
Process
Formulate Problem
Determine Research Design
Design Data Collection
Method and Forms
Design Sample and Collect Data
Analyze and Interpret the Data
Prepare the Research Report
Primary Data: Overview
• Types of Primary Data
–
–
–
–
–
–
–
Demographic / Socioeconomic Characteristics
Psychological / Lifestyle Characteristics
Attitudes / Opinions
Awareness / Knowledge
Intentions
Motivation
Behavior
• What, how much, where, when, how, who
• Purchase behavior vs. use behavior
• E.g., --- basic hierarchy of effects models
Primary Data: Overview
• Qualities of Primary Data
– Versatility
– Can be obtained quickly
– Quality
• Objectivity
• Accuracy
Primary Data: Overview
• Two Classes of Primary Data
– Observation
– Communication
Decisions, Decisions
Communication
Degree of
Structure
Structured
Unstructured
Degree of
Disguise
Degree of
Structure
Disguised
Undisguised
Personal Interview
Telephone Interview
Mail Questionnaire
Structured
Unstructured
Degree of
Disguise
Disguised
Undisguised
Method of
Administration
Observation
Setting
Method of
Administration
Natural
Contrived
Human
Mechanical
Structure and Disguise Decisions
Structured
Unstructured
Undisguised Typical Questionnaire
Interviews
(very frequently used) Open-Ended Questions
Disguised
least used
Motivation Research
Word Association
Sentence Completion
Story Telling
Trade-Offs Among Methods of
Survey Administration
HIGHLY
VERSATILE
LIMITED
phone
fax
mail
email
web
personal
(e.g., mall, home)
LOW
COST
HI COST
personal
phone
fax
mail
web
email
SLOW
mail
personal
fax
web
phone
email
FAST
TURNAROUND
Mall Intercepts vs. E-Panels
household size
average age
employed
white
male
college
mall
tests
2.8
40.5
71%
86%
20%
40%
internet
tests
2.9
39.2
72%
88%
21%
43%
panel
members
3.0
37.2
69%
89%
15%
46%
Correlation between Responses:
purchase intent
frequency
liking
price / value
mall vs.
internet
.86
.94
.85
.90
internet
test/retest reliability
.94
.97
.91
.99
E-Panels vs. Phone
Time survey took to administer
Internet
12.5
Phone
19.4 minutes
Upon completion, would respondent
participate in future studies?
35% yes
26% yes
More experienced Internet Users
x
Used rating scale extreme “endpoints”
more frequently
Jeff Miller and Alan Hogg “Internet vs. Telephone Data Collection” Burke White Paper series 2 (4)
(www.burke.com). Also see Ashok Ranchhod and Fan Zhou “Comparing Respondents of E-Mail and Mail
Surveys,” Marketing Intelligence & Planning 19 (2001), 254.
x
Basic Question Formats
• Open-ended question
• Unprobed format: seeks no additional
information
• Probed format: includes a response
probe instructing the interviewer to
ask for additional information
Basic Question Formats
• Close-ended question
• Dichotomous: has only two response
options, such as “yes” or “no”
• Multiple response: has more than two
options for the response
• Scaled-response: utilizes a scale
developed by the researcher to
measure the attributes of some
construct under study
• Labeled vs. Unlabeled
Other Considerations…
• The nature of the property being
measured
• Previous research studies
• The data collection mode
• The ability of the respondent
• The scale level desired…
Types of Scales
• Nominal scales: those that use only
labels
• Ordinal scales: those with which the
researcher can rank-order the
respondents or responses
• Interval scales: those in which the
distance between each descriptor is
equal
• Ratio scales: ones in which a true zero
exists
Examples…
Nominal
Ordinal
Which of the soft drinks in the
following list do you like?
(Check ALL that apply):
___Coke
___Dr. Pepper
___Mountain Dew
___Pepsi
___Seven Up
___Sprite
Rank the soft drinks according to how much you
like each (most preferred drink = 1, and least
preferred drink = 6):
___Coke
___Dr. Pepper
___Mountain Dew
___Pepsi
___Seven Up
___Sprite
Interval
Ratio
Please indicate how much you like each soft
drink by checking the appropriate position on the
scale:
dislike
like
a lot dislike like
a lot
Coke
____ ____ ____ ___
Dr. Pepper
____ ____ ____ ___
Mountain Dew
____ ____ ____ ___
Pepsi
____ ____ ____ ___
Seven Up
____ ____ ____ ___
Sprite
____ ____ ____ ___
Please divide 100 points among these soft drinks
To represent how much you like each:
___Coke
___Dr. Pepper
___Mountain Dew
___Pepsi
___Seven Up
___Sprite
100
15
Thinking Ahead…
What analysis technique is used when…
1.
2.
3.
4.
5.
There are 2 nominally scaled variables
There are 2 intervally (or ratio) scaled variables
There is one DV that your trying to predict or explain
with multiple interval variables (the independent
variables)
You are checking for differences between 2 groups
(nominal) for an interval scaled variable
You are checking for differences between 4 stores on
total $ sales across 4 categories?
Potential Answers --- t-test; multiple regression, oneway
ANOVA, factorial ANOVA, correlation, crosstabs,
multivariate analysis of variance
Survey (Questionnaire) Design
Procedure for Developing a
Questionnaire
Step 1: Specify What
Information Will be Sought
Step 2: Determine Type of Questionnaire
and Method of Administration
Step 3: Determine Content of
Individual Questions
Step 4: Determine Form of
Response to Each Question
Procedure for Developing a
Questionnaire
Step 5: Wording of
Each Question
Step 6: Sequence
of Questions
Step 7: Physical Characteristics
of Questionnaire
Step 8: Re-examine Steps 1-7
and Revise if Necessary
Step 9: PRE-TEST the Survey,
Revise where Needed
Desirable Characteristics
•
•
•
•
Brief
Objective
Specific
Relevant
“Other” Important Issues
• Respondent qualification
• Frame of reference
• Question loading
QUESTION SEQUENCING General Guidelines
• Use (more) simple, interesting opening
questions
• Use the funnel approach, asking broad
questions first, and follow with more
specific questions
• Carefully design branching questions
• Ask for classification information last
• Place more difficult or sensitive
questions near the end
QUESTION SEQUENCING General Guidelines
Question ordering #1
1 – EVALUATION OF FAT LEVEL OF A PRODUCT
2 – EVALUATION OF OVERALL PROD. NUTRITIOUSNESS
3 – EVALUATION OF OVERALL PRODUCT ATTITUDE
AND INTENTIONS TO PURCHASE
Question ordering #2
1 – EVALUATION OF OVERALL PRODUCT ATTITUDE
AND INTENTIONS TO PURCHASE
2 – EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS
3 – EVALUATION OF FAT LEVEL OF PRODUCT
QUESTION WORDING - General
Guidelines
• Use simple words and questions
• Avoid ambiguous words and
questions
• Avoid leading questions---be
objective
• Avoid implicit alternatives
• Avoid generalizations and
estimates ---Be specific
• Avoid double-barreled questions
1. What is your income?
$10,000 or less………………….1
$10,000 to $25,000……………..2
$25,000 to $50,000……………..3
$50,000 to $75,000………..…….4
$75,000 to $100,000..……..…….5
$100,000 or more…………..……6
What is the problem and how would you
revise the question?
2. Do you agree with many experts that the United
States will run out of oil by the year 2020?
Yes………………….1
No……………...……2
3. How important was price compared to
service when your company decided to buy its
current computer network?
Very important…………….4
Somewhat important……….3
Somewhat unimportant…….2
Very unimportant...…...……1
Please rate the importance of the following factors in
making the choice among vendors.
Selling price:
Not
Very
Important ___ ___ ___ ___ ___ ___ ___ Important
Vendor’s reputation for service:
Not
Very
Important ___ ___ ___ ___ ___ ___ ___ Important
4. Is the speed and efficiency of the drive-in teller
services at your regular bank…..(READ
CATEGORIES)
Very Satisfactory………………………4
Somewhat satisfactory…………………3
Somewhat unsatisfactory………………2
Very unsatisfactory……………….……1
PROBLEMS
a. Speed and efficiency (double-barreled)
b. Do they use drive-in teller service; Do R’s know what drive
in tellers are? What about ATM drive-ins?
c. What are regular banks?
Itemized Rating Scales
• The respondents are provided with a scale that
has a number or brief description associated
with each category.
• The categories are ordered in terms of scale
position, and the respondents are required to
select the specified category that best describes
the object being rated.
• The commonly used itemized rating scales are
the Likert and semantic differential
– Others include Thurstone and Guttman
Itemized Rating Scales
• Likert Scales
– requires the respondents to indicate a degree of
agreement or disagreement with each of a series of
statements about the stimulus objects
Strongly
disagree
Disagree
Neither
agree nor
disagree
Agree
Strongly
agree
1.
Wal-Mart sells high quality merchandise. 1
2X
3
4
5
2.
Wal-Mart has poor in-store service.
1
2X
3
4
5
3.
I like to shop at Wal-Mart .
1
2
3X
4
5
4.
Wal-Mart has low prices .
1
2
3X
4
5
Itemized Rating Scales
• Semantic Differential Scales
– End points associated with bipolar labels that have
semantic meaning
SEARS IS:
Powerful
--:--:--:--:-X-:--:--: Weak
Unreliable --:--:--:--:--:-X-:--: Reliable
Modern
--:--:--:--:--:--:-X-: Oldfashioned
Key Decisions for Itemized Scales
•
Number of categories
•
Balanced vs. unbalanced
•
Odd/even no. of categories
•
Forced vs. non-forced
•
Verbal description
•
•
•
•
•
Although there is no single, optimal number, traditional guidelines
suggest that there should be between five and nine categories
In general, the scale should be balanced to obtain objective data
If a neutral or indifferent scale response is possible for at least some
respondents, an odd number of categories should be used
In situations where the respondents are expected to have no opinion,
the accuracy of the data may be improved by a non-forced scale(
DK/NR)
An argument can be made for labeling all or any scale categories. The
category descriptions should be located as close to the response
categories as possible
Measurement Theory
• Construct Validity
– Criterion (Face) Validity
– Predictive Validity
– Nominological Validity
• Reliability
Construct Validity
Construct
Abstract Concept
“Unobservable”
“Latent”
“Psychological”
We often use Multi-item measures
(scales) to assess unobservable
constructs (WHY???)
Reliability
•
True Score Test Theory
–
All measures have
1.
2.
•
•
“True” Score
“Error” (Random and Systematic)
Good measures minimize the
systematic error component of the
score
Types of Reliability
–
–
–
Inter-Rater
Test-Retest
Internal Consistency (Cronbach’s Alpha)
Developing Sound Measures
Step 1:
Specify Domain
of the Construct
Step 2:
Generate Sample
of Items
Step 3:
Collect Data
Step 4:
Purify Measure
Step 5:
Assess Validity
Tips for Maximizing Participation
•
•
•
•
•
•
•
Personal communication
Asking a favor
Importance of the research project and its
purpose
Importance of the recipient, how recipient
was selected
How the recipient may benefit from this
research
Completing the questionnaire will take only a
short time
The questionnaire can be answered easily
Tips for Maximizing Participation
•
•
•
•
•
•
•
•
A stamped reply envelope is enclosed
Answers are anonymous or
confidential
Offer to send report on results of
survey
Appreciation of sender
Importance of sender or sender’s
organization
Description and purpose of incentive
Style, format and appearance
Brevity
Tips for Maximizing Participation
Professional Mail Surveys Company
7432 East Court Avenue
Elveron, CA 90101
(213) 991-5550
Dear Mr. Smythe:
Will you do us a favor?
We are conducting a nationwide survey among executives and managers in the metalworking
industry. The purpose of this research is to find out the opinion of experts like you on the advantages and
disadvantages of using three new steel products. Your answers will enable steel manufacturers to be aware of
the requirements of the users and the opinions of the nonusers of these products, and this in turn will help
them design the products you need.
Your name appeared in a scientifically selected random sample. Your answers are very important to
the accuracy of our research, whether or not your company is a user of one or more of the products described.
It will take only a short time to answer the simple questions on the enclosed questionnaire and to
return it in the stamped reply envelope.
Of course all answers are confidential and will be used only in combination with those of other
metalworking executives and managers.
If you are interested in receiving a report on the findings of this research, just write your name and
address at the end of this questionnaire, or if you prefer, request the results of the Survey on Steel Products in
a separate letter. We will be glad to send you a complimentary report when it is ready.
Please return the completed questionnaire at your earliest convenience. Thank you for your help!
Sincerely,
James B. Jones
James B. Jones
Director
P.S. The enclosed dollar bill is just a token of appreciation.
And finally, remember the
golden rule….
Do unto your respondents as you
would have them do unto you!!
Team Assignment #2
** Don’t start on this assignment until
you’ve read Fowler (CH 6-7)
• Based on your research design
1. Write a paragraph about what your
measurement instrument is supposed to
accomplish
2. Make a list of what should be measured to
accomplish the goals of the study
3. Develop your measurement instrument
Team Assignment #2
• Deliverables include:
1. A very clean, polished version that you
could use to actually collect data
•
This means you will need to carefully think
through all of the issues we covered tonight
(e.g., set-up, ordering, length, multi-item
scales, etc.)
2. Intro paragraph and variable list (see
previous slide)
(Note: Don’t worry about defining your sample--you’ll
have your chance to do that next week)