Research Method
Step 1 – Formulate research question
Step 2 – Operationalize concepts
◦ Valid and reliable indicators
Step 3 – Decide on sampling technique
◦ Draw sample
Step 4 – Select data collection technique
◦ Collect data
Step 5 – Analyze data
Step 6 – Write up the report
It is critically important to develop valid and
reliable measurements/indicators

If your measurements/indicators are not
valid and reliable then you are wasting
your time. When you input your data,
never forget……………
If you put trash into the
computer, you will get
trash out, no matter how
sophisticated your
analysis.
What is a Valid and Reliable
Measurement?

Validity
◦ Refers to the relationships between a concept and its indicator
◦ Is the indicator an accurate measurement of the concept?

Reliability
◦ Refers to consistency across time and place
◦ Do you get consistent or same results when indicator is used in
different, but comparable, time and/or place?
◦ NOTE – If you don’t get consistent results then it could be that
 Measurement is ambiguous – faulty
 Questions are double barreled or confusing, even in same setting
 Example – Do you agree with the following statement – Men and Women are Good
Communicators? – is a double barreled statement
 Situation is different and measurement doesn’t hold across these situations
 Terms have different meanings in different subcultures
 Example - Do you think it’s BAD to get a tattoo? – is a statement that means something
entirely different to teenagers than to their parents
Four Types of Validity
to Consider

Face Validity
◦ Does indicator “obviously” measure the concept? Is it a
“sensible” indicator?

Content Validity
◦ Does indicator cover entire range of meaning of the concept?
 If concept is multi-dimensional, then is indicator multi-dimensional?

Construct Validity
◦ Is indicator related to other indicators as specified by the
literature?

Criterion Related or Predictive Validity
◦ If the concept is supposed to predict a future event, then does
the indicator predict that same future event accurately?
A More Detailed Look
at Face Validity

Face Validity—Indicator is a sensible or obvious
measurement of the concept
◦ If concept is a type of behavior, then indicator should
measure behavior
 Common mistake—using the number of workers of color hired
by a company as a measure of prejudice
 Hiring is a behavior and prejudice is an attitude
 This would measure discrimination not prejudice
◦ If concept is a value laden concept, then we must take
social desirability into account when constructing a
measure
 Common mistake—measuring crime by asking people if they have
committed a crime
 No one wants to admit they’ve committed a crime
A More Detailed Look
at Content Validity

The indicator must cover the entire range of the meaning
of the concept
Examples
◦ If you measure attitudes toward a workshop, you must ask multiple
questions to cover the multiple aspects of the workshop (i.e., quality of
handouts, quality of presentations, relevancy of information, etc. )
◦ If you measure social class (a multi-dimensional concept) you must
measure income, occupation and education
◦ If you measure prejudice, you must either think about and measure all of
the different types of prejudice (i.e., racial, religious, social class
prejudice) or limit yourself to one type and indicate that when you
discuss your concept
A More Detailed Look
at Construct Validity

Indicator must be related to other indicators and/or
concepts as determined by past research reported in the
literature

Theoretical Construct Validity—Indicator is related to other
concepts/indicators as specified by a theory
Example – As predicted by theory, your indicator of
poverty is related to whether or not they live in a
single parent household
A More Detailed Look
at Construct Validity

Indicator must be related to other indicators and/or
concepts as determined by past research reported in the
literature.

Discriminate Validity—Indicator is related to other indicators,
measurements or behaviors as predicted by the literature or past
research
Example – As predicted in the literature, your volunteers are
happier when they have some “voice” in the decisions that are
made. Your measurements on happiness and decision making
power are related as they should be.
A More Detailed Look
at Construct Validity

Convergent Validity—Indicator is related to data using other data
collection methods as predicted (multi-methods)
Example – When children who attend your workshops and
“appear” to be happier when observed, also score higher on a
happiness measurement.

Known Groups Validity—Indicator is related to groups with known
characteristics as expected
Example – KKK members score higher on prejudice index than
members of civil rights movement.
A More Detailed Look
at Construct Validity

Factor Validity—Indicator is related to other items in same subscale
more strongly than to items in different subscale
Example – the CES-D scale measures 4 components of
depression (negative affect, lack of positive affect, somatic
symptoms and interpersonal). Each of these components is
measured by several items/statements that form a subscale.
To have factor validity, a single item/statement must be more
strongly related to other items in that subscale than to items
in another subscale. For instance, in the negative affect
subscale there are items measuring feeling blue, feeling sad
and feeling depressed. These items are more strongly related
to each other than to items in the somatic symptoms subscale
(i.e., overeating, difficulty concentrating, sleeping too much).
You would use a factor analysis to determine this.
A More Detailed Look at Criterion,
Concurrent or Predictive Validity

Criterion Related Validity—Scores on one indicator can be used to
predict scores on another.
Example - Scores on marital happiness scale can predict
scores on personal happiness scale.

Concurrent Validity—Scores on your indicator can be used to
predict current behavior.
Example – SAT/ACT scores are related to current
performance in school (GPA)
A More Detailed Look at Criterion,
Concurrent or Predictive Validity

Predictive Validity—Indicator can be used to predict
future events
Example - SAT/ACT scores related to
performance in college (GPA)
Reliability

Reliability refers to consistency across time. An indicator can be
reliable (provide consistent results), but NOT valid (accurate). It
can provide consistently WRONG answers.
◦ There are different ways to measure reliability, which include:
 Test/retest
 Internal consistency
 Using alternative forms
 Inter-rater reliability
 Intra-rater reliability
Reliability – Consistency of Indicators

Test/retest
◦ Subjects provide same answers to the same items at different
times
◦ Individuals should score the same each time

Internal consistency
◦ Scale items are highly correlated/associated with each other
◦ Use a Cronbach’s Alpha to determine this.

Alternative forms
◦ Use slightly different forms – see example on next slide

Inter-rater reliability
◦ Two or more researchers get same results

Intra-rater reliability
◦ Same researcher get similar results across time
Using different ways of asking the question
should yield the same answers
SD
D
A
SA
I liked the workshop presentation
1
2
3
4
I like the workshop presentation
SA
1
A
2
D
3
SD
4
I did not like the workshop
presentation
SD
1
D
2
A
3
SA
4
D
2
U
3
A
4
SA
5
I liked the presentation
SD
1
Relationship Between Validity
and Reliability

Definition of terms
◦ Validity – accuracy
◦ Reliability – consistency

Relationships
◦ If it is valid (accurate) then it is reliable
(consistent)
◦ BUT if it is reliable – it may not be valid, it
could be consistently WRONG
Examples – Bathroom Scales

Valid
◦ Scales provide accurate measurement of weight
◦ As long as you don’t gain or lose weight, then
they will also provide consistent weight

Reliable
◦ Scales provide consistent measurement of weight
◦ BUT if you have not calibrated scales accurately,
they may be consistently wrong
Questions or comments?
Please contact:
Carol Albrecht
Assessment Specialist
USU Extension
979-777-2421
carol.albrecht@usu.edu