 What
is reliability?
 What is validity?
 Can you have one without the other? Why or
why not?
X
= True score + random error + systematic
error. Explain. Why is this important?
 How can you decrease these errors?
 What does a reliability estimate of .85 tell
you?
 What do we want our reliability estimates to
be?
 What
are each, how do you calculate them,
when would you use them, and how can you
increase them?



Inter-rater reliability
Test-retest reliability
Parallel-forms reliability
 What
are each, how do you calculate them,
what do they tell you, where do you want
values to be, and how can you increase
them?





Average inter-item correlation
Average item-total
Split-half reliability
Cronbach’s alpha
Kuder-Richardson Formula 20 (KR-20)
SPSS printout
 How
does construct validity relate to internal
and external validity?
 What are each of these, how would you
calculate them, and what do they tell you?







Face validity
Content validity
Criterion-related validity
Predictive validity
Concurrent validity (aka known groups)
Convergent validity
Discriminant validity/Divergent validity
 How
high/low should your correlations be?
 Nomological
network
 Multitrait-multimethod matrix (MTMM)


Look at example p. 70
What information does this give you?
 Pattern


matching
Advantages/disadvantages
How can SEM be used to show this?
 What
are these, and how can the problem be
decreased?







Inadequate preoperational explication of
constructs
Mono-operation bias
Mono-method bias
Interaction of different treatments
Interaction of testing and treatment
Restricted generalizability across constructs
Confounding constructs and levels of constructs
 What
are these, and how can the problem be
decreased?



Hypothesis guessing
Evaluation apprehension
Experimenter expectancies
 Other



threats
Social desirability
Response styles
Demand characteristics
 Podsakoff,
 What
MacKenzie, Podsakoff, 2012
is it?
 Is it a problem?
 What causes it?
 What effects does it have?
Use more than one method—get predictor and
criterion from different sources
 Separate measures temporally, proximally, or
psychologically
 Use different types and points on scales
 Make items less ambiguous
 Decrease tendencies for socially desirable
responses
 Reverse score items
 Try to increase motivation and ability of
participants
 Make questions simple and easy to follow
 Control for bias statistically (several options)

 Why
do they matter?
 What are the four types? Examples? Stats
that can be run on them?




Nominal
Ordinal
Interval
Ratio
 What
types of scales do we typically use in
psychology? Is that a problem?
 Conceptualize
the index
 Operationalize and measure the components
 Develop the rules for calculating the index
score (weighting?)
 Validate it!
 What
are unidimensional vs.
multidimensional scales? When should you
use one vs. the other?
 Scaling vs. response scales (every set of
questions is not a scale)
 What is a semantic differential? What would
be good items?
 Thurstone
(method of equally appearing
intervals)



Generate items
Have judges rate them
Choose ones that represent the whole scale
 Guttman

(cumulative scale)
Coefficient of reproducibility
 Likert
(summative rating scale)
 What are the advantages and disadvantages
of each?
 How do you score each?
I believe the church is the greatest institution in
America today. (.2)
 I believe in religion, but I seldom go to church.
(5.4)
 I believe in sincerity and goodness without any
church ceremonies. (6.7)
 I believe the church is a hindrance to religion for
it still depends on magic, superstition, and myth.
(9.6)
 I think the church is a parasite on society. (11.0)

I
I
I
I
I
I
I

am
am
am
am
am
am
am
more
more
more
more
more
more
more
than
than
than
than
than
than
than
54
56
58
60
62
64
66
inches
inches
inches
inches
inches
inches
inches
tall.
tall.
tall.
tall.
tall.
tall.
tall.
SD D U A SA 1. The federal government has
no business supporting child care.
 SD D U A SA 2. Day care is an issue that the
government should fully support
 SD D U A SA 3. Tax money should be used to
fund day care programs.

 Standardization
 Norms






Find an article that describe the development of a
scale related to your thesis topic (should include
multiple studies, assessing reliability and validity).
Evaluate the article and scale (and include a brief
description of the scale, including the APA style
citation).
How did they show the various types of reliability
and validity? Do you agree?
What type of scaling is the measure? What type of
scale?
Are there similar concepts that still need a scale
developed? How would you go about creating a scale?
What would you correlate it with to test reliability
and validity?
What other comments or questions do you have on
reliability/validity/scaling?
 To
choosing a scale to use?
 To creating a scale?
Comments on 3 articles and the chapter PLUS
 Find examples of “bad” survey questions. Think
about any issues that you might need to deal
with in terms of how you ask questions on your
thesis. What types of issues (e.g., population,
sampling) might come up and how would those
affect what method you choose? What are the
advantages and disadvantages of survey
methodology for your topic? What other
comments or questions do you have based on
these readings?
 Outlines of intro for paper due today
