ScWk 240
Week 6
Measurement Error
Introduction to Survey Development
“England and America are two countries
divided by a common language.”
George Bernard Shaw
ScWk 240 Week 6
1
Review--Variables:
• When we use the word “outcome” are we talking
about the independent variable or the dependent
variable?___________
• In an evaluative study, the primary independent
variable is the __________
• “This study examines the impact of acculturation on
the use of health services by immigrants, controlling
for income, age, and health status.
– What is the dependent variable? ___________
– What is (are) the independent variable(s)?
_______________________________________________________________
ScWk 240 Week 6
2
Review: Conceptualization
• We’re talking about complex ideas
– Depression
– Self esteem
– Acculturation
• How would you conceptualize
“acculturation”?
– Give a one sentence conceptual definition
– What dimensions would you include?
ScWk 240 Week 6
3
Operationalization – the next step
• “Operationalization” – the process of defining a specific
variable, and identifying the relevant attributes of the
variable
• Example – Depression “loss of energy” dimension
– Variable: “loss of energy”
– Measured by: (CIDI instrument) During the period when
your sadness was most severe and frequent, did you feel
tired or low in energy even when you had not been
working very hard?
– Attributes:
– Yes
– No
– Don’t know
ScWk 240 Week 6
4
Now operationalize “acculturation”
• Pick one dimension
• For that dimension, describe the variable in a
simple statement
– First, give the variable a label (name) in 1-3 words
– Include in the simple statement “…as measured
by…”
• For that variable, identify one set of attributes
ScWk 240 Week 6
5
Small group exercise:
•
•
•
•
Pick a concept as a group
Identify three dimensions of that concept
Choose one dimension
For that dimension, identify and describe one
variable in a simple statement
– First, give the variable a label (name) in 1-3 words
– Include in the simple statement “…as measured
by…”
• For that variable, identify one set of attributes
ScWk 240 Week 6
6
Student Report
What is “culturally competent measurement” in
research? Relate this to one or more (of the 5)
areas of the Transcultural Perspective. Elisse
ScWk 240 Week 6
7
“Error” in Measurement
• What is “measurement error”?
Measurement error occurs when we obtain data that do not
accurately reflect the concept we want to measure.
• There are two main sources of measurement
error:
– Random error: inconsistent error. There’s no
pattern or predictability
– Systematic error: consistent error. There is pattern
and predictability
ScWk 240 Week 6
8
Random Error – inconsistent response
– Environmental factors: i.e. surveys administered in
a hot room will vary from surveys administered in
a more comfortable setting
– Questions with jargon or technical terms will be
interpreted differently by different respondents
(or other factors related to a poorly constructed
instrument)
– Written survey response will differ by educational
level or reading ability of respondents
ScWk 240 Week 6
9
Systematic error—consistently and
predictably “wrong” response
• Social desirability bias’ – respondents portray
themselves in the best light
• ‘Acquiescent response set’ – Agree or disagree with
everything
• Cultural ‘bias’ – cultural definitions of concepts
different than those intended by researcher (i.e. you
can always expect physical somatization as a stated
primary symptom of depression in many nonWestern cultures)
• Bias from wording of questions – i.e. “Are you
resistant to treatment?”
ScWk 240 Week 6
10
Review: Potential for systematic or
random error?
1. People are more alert in the morning than after
lunch ____________
2. The meaning of “living independently” varies crossculturally __________
3. Male respondents tend to minimize their aggressive
tendencies _________
4. A 16 y.o. taking a test: “I just circle ‘C’ each time
and hope for the best” (hint-this is not Acquiescent
Response) _________
5. The person entering data is not careful enough and
mis-codes some of the responses ________
ScWk 240 Week 6
11
Student reports
1. What’s the difference between “reliability”
and “validity” of a measure? Glorialee
2. How is validity related to cultural issues?
Dawei
ScWk 240 Week 6
12
Measuring error—Validity and
reliability
• The presence of error is not “all or nothing” –
there’s always going to be some error
• We have to measure it to understand its
impact
• Reliability – measures random error—
consistency of meaning
• Validity – measures systematic error—
accuracy of meaning
ScWk 240 Week 6
13
Reliability
• Concerned with whether or not a particular
technique done repeatedly would yield the
same result each time
– Inter-rater reliability—Two researchers rating the same
behavior: do they rate them the same?
– Test-retest reliability—Is the response from the first test
the same as the response from a re-test?
– Alternate forms reliability—Is there the same response
from slightly different versions of same measure?
– Internal consistency reliability & use of coefficient alpha—
Are similar questions (about the same concept) in survey
are answered in the same way?
ScWk 240 Week 6
14
You‘ll see this in your lit review…
• Methods section—measures used: the author
will report on reliability testing (if done) using
the coefficient alpha
– A number that tells you, overall, how well the
questions in an instrument measuring the same
concepts correlate with each other (internal
consistency reliability)
– A coefficient alpha is considered good if it is .80 or
above
– The author will then report “The instrument
showed good internal consistency reliability”
ScWk 240 Week 6
15
Validity
• Concerned with the accuracy of meaning—are
you measuring what you originally intended?
• Types of validity
– Face validity – experts’ understanding of concept
– Content validity – Does measure cover all relevant
dimensions?
– Criterion-related validity – Does measure of concept agree
with (or relate with) another related concept (external
criterion)?
– Construct validity – Does measure agree with or correlate
with measures of other theoretically related concepts?
ScWk 240 Week 6
16
Validity examples
1. Validity studies show that a set of questions about alcohol
dependence correlates well with the observed daily amount
of alcohol consumption. Criterion-related validity
2. Most experts would agree that children with Attention
Deficit Disorder with Hyperactivity have difficulty staying on
tasks (even those the child likes). Face validity
3. Couples who test as having low levels of marital satisfaction
communicate less effectively, leading to high levels of conflict
and, eventually, divorce. Construct validity
4. The Beck Depression Scale covers those dimensions of
depression that are most supported by evidence. Content
validity
ScWk 240 Week 6
17
Student reports for next week
• 1. Dawei
• What are some strategies for ensuring
reliability when developing survey
questions?___Amanda
• What are some strategies for ensuring validity
when developing survey questions?____Sarah
• What is the main difference between “openende” and “closed-ended” questions? Selene
ScWk 240 Week 6
18
Concepts to know
•
•
•
•
Measurement error
Random error
Systematic error
Reliability (Inter-rater, Test-retest, Alternate
forms, Internal consistency)
• Coefficient alpha
• Validity (Face, Content, Criterion-related,
Construct)
ScWk 240 Week 6
19