Quantitative
Research
March 8
Quiz
 Assignments
 Experimental Research
 Quantitative Research
 Emotional Intelligence Survey
 Validity, Reliability, Error
 Characteristics of Surveys

Experimental Research
Laboratory Experiments






Artificial – low realism
Few extraneous variables
High control
Low cost
Short Duration
Subjects aware of
participation
Field Experiments






Natural – high realism
Many extraneous variables
Low control
High cost
Long Duration
Subjects unaware of
participation
Quantitative Research

Emotional Intelligence Survey
http://quiz.ivillage.co.uk/uk_work/tests/eqtest.htm
Validity
Extent to which a measurement instrument
actually measures the attribute it was
intended to measure.
 Validity can be examined from a number of
different perspectives, including:

 Face,
content, criterion-related, and construct
validity
Assessing Validity
Face validity
Researchers judge the degree to which a measurement
instrument seems to measure what it is supposed to.
Content validity
The degree to which the instrument items represent
the universe of the concept under study.
Criterion-related
validity
The degree to which a measurement instrument can
predict a variable that is designated a criterion.
(a) Predictive ability; (b) Concurrent validity
Construct validity
The degree to which a measure confirm a hypothesis
created from a theory based upon the concepts under
study.
(a) Convergent validity; (b) Discriminate validity
Reliability

Reliability
 Degree
to which measures are free from
random error and, therefore, provide
consistent data.

There are three ways to assess reliability
 Test-retest,
equivalent forms, and internal
consistency
Assessing the Reliability of a
Measurement Instrument
Test-retest reliability:
Use the same instrument a second time under
as nearly the same conditions as possible.
Equivalent form reliability:
Use two instruments that are as similar as
possible to measure the same object during the
same time period.
Internal consistency reliability:
Compare different samples of items being
used to measure a phenomenon during the
same time period.
Reliability and Validity Situations
in Measurement
Situation 1
.
.
.
.
.
.
Situation 2
.
Situation 3
.........
. .
.
.. ..
..
.
.
Neither reliable
nor Valid
Highly reliable
but not valid
Highly reliable
and valid
Error
Types of Error in Survey
Research

Random Sampling Error (Random error)
 Error
that results from chance variation
 Impact can be decreased by increasing
sample size and through statistical
estimation (confidence interval) or “rule of
thumb”

Systematic Error (non sampling error)
 Error
that results for the research design or
execution.
Total Error
Random Error
Systematic Error
Respondent error
Administrative error
Non response error
Response bias
Deliberate Falsification
Acquiescence
Extremity bias
Unconscious misrepresentation
Interviewer bias
Auspices bias
social desirability
Data Processing
Sample selection
Interviewer cheating
Interviewer error
Types of Systematic Error
1. Administrative Error
 Error

that results from improper execution.
Data Processing Error
 Quality
of data depends on quality of data
entry.
 Use of verification procedures can
minimize

Sample Selection Error
 Systematic
error resulting from improper
sampling techniques either in design or
execution.

Interviewer Error
 Data
recorded incorrectly (error or
selective perception).

Interviewer Cheating
 Mitigate
by random checks
2. Respondent Error
 Humans

interviewing humans...
Non-response error
 Statistical
difference between a survey that
includes only those who responded and a
survey that also includes those who failed to
respond.
 Non-respondent: person not contacted or who
refuses to participate
 Self selection bias: extreme positions
represented

Response bias
 Errors
that result from tendency to answer in
“a certain direction”.
 Conscious or unconscious misrepresentation

Types:
 1.
Deliberate falsification (why?)

Why would people deliberately falsify data
 Appear
to be what they are not
 Don’t trust confidentiality
 Protect
 To end the interviewer quicker
 “Average man effects”

Types of response bias continued:
 1.
Deliberate falsification
 2. Unconscious misrepresentation

Reasons for unconscious
misrepresentation:
 Question
format
 Question content
 Misunderstanding of question leading to
biased answer
 Lack of time to consider answer fully
 Communication or semantic confusion
 other
Types of response bias
 Acquiescence bias: individuals have a
tendency to agree or disagree with all
questions or to indicate a positive/negative
connotation
 Extremity bias: results for response styles
varying from person to person; some
people tend to use extremes when
responding to questions
Types of response bias continued...
 Interviewer bias: Bias in the responses of
the subject due to the influence of the
interviewer
 Auspices bias: respondents being
influenced by the organization conducting
the study
 Social desirability bias: caused by
respondents’ desire, either consciously or
unconsciously to gain prestige or to
appear in a different social role
Measurement
The Concept of Measurement
and Measurement Scales
• Measurement
• Process of assigning numbers or labels to things
in accordance with specific rules to represent
quantities or qualities of attributes.
• Rule: A guide, method, or command that tells a
researcher what to do.
• Scale: A set of symbols or numbers constructed to
be assigned by a rule to the individuals (or their
behaviors or attitudes) to whom the scale is applied.
Types of Measurement Scales
• Nominal Scales
• Scales that partition data into mutually
exclusive and collectively exhaustive
categories.
• Ordinal Scales
• Nominal scales that can order data.
• Interval Scales
• Ordinal scales with equal intervals between
points to show relative amounts; may include
an arbitrary zero point.
• Ratio Scales
• Interval scales with a meaningful zero point so
that magnitudes can be compared
arithmetically.
Nominal
Win
Place
Show
Ordinal
Interval
1 length
2 lengths
Ratio
40 to 1 long-shot pays $40
Type of Scale
Nominal
Numerical
Operation
Counting
Descriptive
Statistics
Frequency;
Percentage;
mode
(plus…)Median
Range; Percentile
Ordinal
Rank ordering
Interval
Arithmetic
operations on
intervals bet
numbers
(plus…) Mean;
Standard
deviation;
variance
Ratio
Arithmetic
operations on
actual quantities
(plus…)
Geometric mean;
Co-efficent of
variation
Selecting appropriate univariate
statistical method
Scale
Nominal
Scale
Business
Problem
Identify sex
of key
executives
Statistical
question to
be asked
Possible test
of statistical
significance
Is the number Chi-square
of female
test
executives
equal to the
number of
males
executives?
Scale
Nominal
Scale
Business
Problem
Indicate
percentage of
key
executives
who are male
Statistical
question to
be asked
Possible test
of statistical
significance
Is the
T-test
proportion of
male
executives
the same as
the
hypothesized
proportion?
Scale
Ordinal
scale
Business
Problem
Compare
actual and
expected
evaluations
Statistical
question to
be asked
Possible test
of statistical
significance
Does the
Chi-square
distribution
test
of scores for
a scale with
categories of
poor,good,
excellent
differ from
an expected
distribution?
Scale
Interval or
Ratio scale
Business
Problem
Statistical
question to
be asked
Compare
actual and
hypothetical
values of
average
salary
Is the
sample
mean
significantly
different
from the
hypothesize
d population
mean?
Possible test
of statistical
significance
Z-test
(sample is
large)
T-test
(sample is
small)
Surveys
Survey data
Survey data are obtained when individuals
respond to questions asked by
interviewers or when the individual
responds to questions that he has read
 Quantitative and qualitative information
 Census or sample?

Survey Sampling

Sample Survey
 Sample
of the population
(e.g., sample of Alberta CEOs)

Census Survey
 Complete
population
(e.g., all CEOs in Lethbridge)
Functions
Descriptive
 Causal (limited function)
 Exploratory (limited function)

Information Provided by
Surveys

respondent’s knowledge of facts
 how many times a month do you buy cookies?

respondent’s attitudes
 “chocolate

May describe processes undertaken by
respondent
 Stages

chip is my favorite variety of cookie.”
in the purchase decision, for example
usually self-report data
The type of information gathered
depends on a survey’s objectives.
 Surveys typically have multiple
objectives

Classifications

Method of Communication
 Telephone
 Mail
 Personal

interview
Degree of Structure
 Unstructured
 structured

Degree of Disguise
Limitations:
Structure and disguise
are not clear categories;
most surveys are hybrids

Temporal classifications
 Cross
sectional studies
 Longitudinal studies

Trend Studies


Cohort studies (tracking studies)


Samples general population at each point; complete
turnover in who is actually sampled
Samples from one group over time; e.g., sample of a
graduating class
Panel studies

sample cohort, same specific respondents each sample
Method of Communication
Media
 Personal Interviews
 Telephone Interviews
 Self administered Questionnaires

Questionnaire
design
A survey is only as good as
the questions it asks
What should you ask?
The questions asked are a function of
previous decisions
 The questions asked are a function of
future decisions (such as statistical
analysis)

Key criteria

Questionnaire relevancy
 No
unnecessary information is collected and
only information needed to solve the problem
is obtained. Be specific about your data
needs; tie each question to an objective

Questionnaire accuracy
 Information
is both reliable and valid
Phrasing Questions

Open ended response versus fixed
alternative questions
“?”

Decision criteria: type of research; time;
method of delivery; budget; concerns
regarding researcher bias
Avoid







Leading questions
Overly complex questions
Use of jargon
Loaded questions (can use a counterbiasing
statement)
Ambiguity
Double barreled questions
Making assumptions
Order?

Order bias results from an alternative
answer’s position in a set of answers or
from the sequencing of questions
 Funneling
technique: general to specific helps
understand the frame of reference first

Anchoring effect: the first concept
measured tends to become a comparison
point from which subsequent evaluations
are made
Types of questions
Types of fixed alternative questions…
• Single dichotomy or dichotomous-alternative
questions
“Are you currently registered in a course at the
University of Lethbridge?
Yes____ No____”
• Respondent chooses one of two alternatives
(yes/no; male/female)
• What scale would this data create?
Types of fixed alternative questions…
 Multi-choice
alternative
Respondent
chooses from several
alternatives
Many types…
Multi-choice alternative questions…

Determinant choice
 Choose
only one from several possible responses
“Which faculty are you currently registered in at
the University of Lethbridge?
Management ___
Education ____
Arts/Science____
Health sciences____
Combined degree____

What type of scale would these data create?

Frequency determination
Asks for an answer about frequency
of occurrence
In a typical week, how often do you
purchase chocolate chip cookies?
__never
__ once
__ 2 or more times
What type of scale would these data create?

Check list
 Provide
multiple answers to a single
question
 Should be mutually exclusive and
exhaustive
“What brands of chocolate chip cookies have
you, to the best of your memory, purchased
in the past month (check all that apply?)”
__ Dare
__ Chips A’hoy
__ Presidents Choice Decadent etc. etc.

What type of scale would these data create?
Unidimensional
Scaling
Multidimensional
Scaling
Procedures
designed to
measure only one
attribute of a
respondent or
object
Procedures
designed to measure
several dimensions
of a respondent or
object
 Attitude
rating scales
Attitude:
An enduring disposition to consistently
respond to various aspect of the
world, including persons, events and
objects
Typically seen as having three
components:
Cognitive
Affective
Behavioural
Affective
The feelings or
emotions toward an
object
Cognitive

Knowledge and beliefs
Behavioral
Predisposition to action
 Intentions
 Behavioral expectations

Attitude measuring process
Ranking
 Rating
 Sorting
 Choice

Types of attitude scales



Simple attitude scales
Most basic form – respondent responds to a
single question
Do not allow for fine distinctions or placement on
continua
 You
are at a company party and are feeling nervous,
but you are obligated to be there. Do you:
__ find someone you know to buddy up with
__ take it as an opportunity to meet new people
What type of scale would these data create?

Category scales
 More
sensitive; provides more information
 Overall, how satisfied are you with the high speed
performance of your Mercedes:
__ very satisfied
__ somewhat satisfied
__ neither satisfied nor dissatisfied
__ somewhat dissatisfied
__ very dissatisfied
If you could choose, how long would each term be?
___26 weeks __ 13 weeks __ 6 weeks ___4
weeks
What type of scale would these data create?

Summated rating scales – the Likert
scale
 Respondents
indicate their attitudes by
checking how strongly they agree or
disagree with statements
 Chocolate
chip cookies are my preferred
variety of cookie
Strongly disagree Disagree Uncertain Agree Strongly Agree
(1)
(2)
(3)
(4)
(5)
What type of scale would these data create?

Semantic Differential Rating scale
 An
attitude measure consisting of a series
of seven-point bipolar rating scales
allowing response to a “concept”
Think of your favorite type of cookie. Rate it on each
of the following continua:
Hard------------------------------------------------------Soft
Lots of chips---------------------------------------Fewer
chips
Crispy---------------------------------------------------chewy
What type of scale would these data create?

Numerical Rating scale
 Similar
to a semantic differential except that it
uses numbers as response options to identify
response positions instead of verbal descriptions
Think of your favorite type of cookie. Rate it on each of the
following continua:
Hard------------------------------------------------------------------------Soft
8
7
6
5
4
3
2
1
This scale is called an 8 point numerical scale, why?
What type of scale would these data create?

Constant Sum Scales
 Attributes
based on their importance to the
person. Respondents are asked to divide a
constant sum to indicate the relative importance of
attributes
Example: Suppose the photocopy budget per
professor was $100 per month. How much should
be allocated to the following. Divide the $100
according to your preference:
____ photocopying for student needs;
____ photocopying for research needs;
____ photocopying for committee needs.
====
$100 TOTAL

Stapel Scales
 An
attitude measure that places a single adjective
in the center of an even-number range of
numerical values
Example:
Research Methodology
+3
+2
+1
Exciting
-1
-2
-3

Graphic Rating Scales
 An
attitude measure consisting of a graphic
continuum that allows respondents to rate
an object by choosing any point on the
continuum

Rank-Order Scales
 Scales
in which the respondent compares
one item with another or a group of items
against each other and ranks them.
Example: handout
Most important skills











Adaptability to change
Problem identification
Listening skills
Written communication
Leadership
Informal Oral communication
Analytical thinking/problem solving
Time management
Coping with stress/job pressures
Interpersonal relations
Formal oral presentations
Most important skills
Adaptability to change
Problem identification
Listening skills
Written communication
Leadership
Informal Oral communication
Analytical thinking/problem solving
Time management
Coping with stress/job pressures
Interpersonal relations
Formal oral presentations
8
6
1
2
4
3
5
7
11
9
10
9
6
1
4
2
3
5
10
7
8
11

Paired Comparison Scales
 Respondent
is presented with two objects
and is asked to pick the preferred.
Example: Which type of cookie do you prefer
__ chocolate chip
__ oatmeal
__ I do not have a preference between these two

Sorting
 Respondent
indicates their attitudes or
beliefs by arranging items.
Example: Please sort the following cards with
pictures of cookies into the following categories
Like
Dislike
Neither like nor dislike