Reliability vs. Validity
One thing researchers are interested in assessing is the _________________________ of their study.
Reliability is a concern to researchers every time an observer is the source of data because there is no
guard from the impact of the observer’s subjectivity (Babbie 2011). Reliability is the extent to which a
survey or experiment will yield the _______________ result on repeated trials. For example, suppose
you asked the following two questions.

“Did you attend religious services last week?”

“About how many times have you attended religious services in your life?”
Question: Which question do you think would yield the same/very similar results on repeated trials?
The big question is then, how do you create a reliable study/survey? Researchers have created several
methods for cross-checking the reliability of measures they devise.
Test-Retest Method: This method involves taking the same measurement __________________
once.
o
Example: 207 subjects were asked to fill out a baseline questionnaire about their
characteristics and behaviors. Then three month later the subjects were given the same
questionnaire again. Only, 15% of the subjects reported the same information in both
surveys.
o
If you don’t expect the information being sought to change, then you should expect the
same response both times.
o
This method is usually quantified by the correlation coefficient.
o
You’ll want to do this over a short amount of time to avoid drastic changes in variables
likely to change in small amounts of time such as happiness.
o
Potential problem – respondents try to familiarize themselves with the questions and
recall answers given the first time.
Split-Half Method: This method involves creating numerous questions which you believe
measure the same thing. Then you split the questions equally between two sets of questions.
The two sets of questions should classify respondents in the same manner. If not, then the
questions are not reliable for that variable.
1
Using Established Measures: This involves using measures which have been proven to be
reliable in previous research.
Reliability of Research Workers: Unreliability can come from research workers through
interviewers or coders.
o
One way to assess reliability is to have a supervisor or other researcher call some of the
same individuals who’ve already been interviewed and verify pieces of information.
o
I you’re worried about how things are being coded, you could have a number of coders
code each item independently and then assess how many were not coded consistently.
These items could then be re-evaluated to determine just where they belong.
o
Clarity, specificity, training, and practice can prevent a great deal of worker unreliability.
Cronbach’s alpha: Measures internal consistency/reliability among groups items combined to
form a single scale.
o
This is a reflection of how well the different items complement each other in their
measurement of different aspects of the same variable.
o
The interpretation is similar to that of the correlation coefficient.
2
Reliability most often gets confused with __________________. Validity, on the other hand is a
measure which determines whether a study/survey is measuring what it is supposed to. For example, a
measure or social class should measure just that and not another measure such as political orientation.
However, the validity of a measure can never be proven per se. There are mechanisms in place for
determining the relative validity of a survey/study on the basis of the following types of validity
measures.
Face validity: This type of validity is concerned with how a measure or procedure appears.
o
Example: You are interested in measuring a person’s IQ. We might not all agree on if
that can adequately measure a person’s IQ , but we’d all agree it is an appropriate
measure.
o
Example: Suppose we are interested in measuring the morale of workers at a particular
company. One way to do this might be to look at the number of grievances that have
been filed. Some might argue that isn’t the best measure, but most people would be in
agreement that it is not totally off base.
Criterion-related (predictive) validity: This type of validity is based on external criterion.
o
Example: College Board exams have been shown to their ability to predict a student’s
success in college.
o
Example: A driving test is used to show a person’s driving ability and how capable they
are of operating a motor vehicle.
Question: Come up with a variable which might be used to validate each of the following attitudes:
a. Supports equality of men and women
b. Is concerned about the environment
c. Is very religious
Construct validity: Sometimes it is hard to find behavioral criteria that can be used as direct
validation tools. We could however approximate such criteria by applying a different test. This
can also be thought of as the degree to which a measure relates to other variables as expected
with a system of theoretical relationships.
3
Content validity: This refers to how much a measure covers the range of meanings included
within a concept.
o
Example: A test of mathematical ability can’t be limited to addition and subtraction, but
also needs to cover other topics such as multiplication and division.
o
Example: Suppose you are interested in conducting a study about prejudice. The
questions created will need to relate to all types of prejudice, not just one or two.
Reliable, not valid
Valid, not reliable
Valid and reliable
4