Housekeeping
• Assignment 1
• Readings for next week:
– Base paper on validity
– A paper from your research area that discusses
validity of an empirical study
Research Design
•
•
•
•
Basic issues of research design
Role of statistics in behavioral research
Classification of variables
Quantification of variables (scales of
measurement)
• Validity of interpretations of research
studies
• Limited to measurement reliability and validity
today
Research Design Decisions
• What kinds of subjects/participants and how
many?
• What will subjects be asked to do?
• How many comparison groups if any?
• What dependent/independent variables to
focus on?
• How and when subjects will be measured?
• Where study will be conducted?
Systems Research Design
Decisions
•
•
•
•
What kinds of data and how much?
What will the trials be?
Under what conditions?
What dependent/independent variables to
focus on?
• How and when will data be measured?
• What will be the study environment?
Design Issues: Participants
• How were they recruited?
• What kinds of population samples?
• How many of intended participants actually
supplied data? Were in final analysis?
• If comparison groups, how were they
formed?
• How motivated were they?
Design Issues: Participants
• How were they recruited?
• What kinds of population samples?
• How many of intended participants actually
supplied data? Were in final analysis?
• If comparison groups, how were they formed?
• How motivated were they?
• What kind of issues are there for systems?
Design Issues: Data
•
•
•
•
Instrument quality
Question/data match
Independence of observations
Person/people collecting data
Design Issues: Study Context
•
•
•
•
•
Physical setting
Pretest sensitization
Treatment conditions
Subjects thoughts about the study
Temporal changes
Descriptive and Inferential
Statistics
• Descriptive Statistics: Methods used to obtain
indices that characterize or summarize data collected
• Inferential Statistics: Methods that allow the
researcher to make inferences from a set of data
collected from a sample to a larger population.
Review of Terms
• Research: a systematic approach to finding
answers to questions.
• Research Design: a plan for gathering data
for answering specific research questions.
• Statistics: the methods used on the data
collected to answer the research questions at
hand.
Basic Elements: Hypotheses
• Hypothesis: a tentative statement
(“educated guess”) about the expected
relationship between two or more variables.
– State expected relationship or difference
between 2 variables
– Be worthy of being tested
– Be testable
– Be brief and clear
Basic Elements: Variables
• Variable: what is measured or varied. An
attribute or characteristic of a person (or object)
that can change from person to person.
–
–
–
–
Independent
Dependent
Control
Intervening (mediator)
Classification of Variables
• Independent Variable: a variable that is
manipulated, measured or selected by the
researcher in order to observe its relation to the
subject's "response” on another variable. An
antecedent condition.
• Dependent Variable: the variable that is observed
and measured in response to an independent
variable.
Classification of Variables (Con’t)
• Control Variable: any variable that is held
constant in a research study by observing only one
if its instances or levels.
• Intervening Variable: a hypothetical variable
that is not observed directly in the research study,
but is inferred from the relationship between the
independent and dependent variable.
• Moderator Variable: Variables that may
moderate the relationship between IV and DV
(gender, race, etc.)
Quantification of Variables
• Measurement: systematic, replicable
process by which objects or events are
quantified and/or classified with respect to a
particular dimension
– Usually achieved by the assignment of
numerical values
• Four (4) scales of measurement
The following slides are from the set provided for:
Measurement: Scaling,
Reliability, Validity
CHAPTER 7
Research Methods in Psychology (6th
Ed.), by Elmes, Kantowitz, &
Roediger
16
Scale
• Is a tool or mechanism by which individuals
are distinguished as to how they differ from
one another on the variables of interest to
our study.
3
scales
•
1.
2.
3.
4.
There are four basic types of scales:
Nominal Scale
Ordinal Scale
Interval Scale
Ratio Scale
18
scales
• The degree of sophistication to which the
scales are fine-tuned increases
progressively as we move from the
nominal to the ratio scale.
• The information on the variables can be
obtained in greater detail when we employ
an interval or a ratio scale than the other
two scales.
19
scales
• With more powerful scales, increasingly
sophisticated data analyses can be
performed, which in turn, means that more
meaningful answers can be found to our
research questions.
20
Nominal Scale
• A nominal scale is one that allows the researcher to assign subjects to
certain categories or groups.
• What is your department?
O Marketing
O Maintenance
O Finance
O Production O Servicing
O Personnel
O Sales
O Public Relations
O Accounting
• What is your gender?
O Male
O Female
7
Nominal Scale
• For example, the variable of gender,
respondents can be grouped into two
categories- male and female.
• Notice that there are no third category into
which respondents would normally fall.
22
Nominal Scale
• The information that can be generated from
nominal scaling is to calculate the
percentage (or frequency) of males and
females in our sample of respondents.
23
Example 1
• Nominally scale the nationality of individuals in a
group of tourists to a country during a certain year.
• We could nominally scale this variable in the
following mutually exclusive and collectively
exhaustive categories.
American
Japanese
Russian
Malaysian
Chinese
German
Arabian
Other
24
Example 1
• Note that every respondent has to fit into
one of the above categories and that the
scale will allow computation of the
numbers and percentages of respondents
that fit into them.
25
Ordinal Scale
• Ordinal scale: not only categorizes variables in such a
way as to denote differences among various categories, it
also rank-orders categories in some meaningful way.
• What is the highest level of education you have
completed?
O Less than High School
O High School/GED Equivalent
O College Degree
O Masters Degree
O Doctoral Degree
26
Ordinal Scale
• The preference would be ranked ( from best
to worse; or from first to last) and numbered
as 1, 2, 3, and so on.
27
Example 2
• Rank the following five characteristics in
a job in terms of how important they are for
you.
You should rank the most important item
as 1, the next in importance a 2, and so on,
until you have ranked each of them 1, 2, 3,
4, or 5.
28
Example 2 (Cont.)
• Job Characteristic
Ranking
The opportunity provided by the job to:
1. Interacts with others
_____
2. Use different skills
_____
3. Complete a task to the end
_____
4. Serve others
_____
5. Work independently
_____
29
Example 2 (Cont.)
• This scale helps the researcher to determine
the percentage of respondents who
consider interaction with others as most
important, those who consider using a
number of skills as most important, and so
on. Such knowledge might help in
designing jobs that would be seen as most
enriched by the majority of the employees.
30
Example 2 (Cont.)
• We can see that the ordinal scale provides
more information than the nominal scale.
Even though differences in the ranking of
objects, persons are clearly known, we do
not know their magnitude.
• This deficiency is overcome by interval
scaling.
31
Interval Scale

Interval scale: whereas the nominal scale
allows us only to qualitatively distinguish
groups by categorizing them into mutually
exclusive and collectively exhaustive sets,
and the ordinal scale to rank-order the
preferences, the interval scale lets us
measure the distance between any two
points on the scale.
32
Interval scale
© 2009 John Wiley & Sons Ltd.
www.wileyeurope.com/college/sekaran
33

Circle the number that represents your feelings at this particular
moment best. There are no right or wrong answers. Please answer
every question.
1. I invest more in my work than I get out of it
I disagree completely
1 2 3 4 5
I agree completely
2. I exert myself too much considering what I get back in return
I disagree completely
1 2 3 4 5
I agree completely
3. For the efforts I put into the organization, I get much in return
I disagree completely
1
2
3
4
5
I agree completely
34
34
• Suppose that the employees circle the numbers 3,
1, 2, 4, and 5 for the five items.
• The magnitude of difference represented by the
space between points 1 and 2 on the scale is the
same as the magnitude of difference represented
by the space between points 4 and 5, or between
any other two points. Any number can be added to
or subtracted from the numbers on the scale, still
retaining the magnitude of the difference.
35
• If we add 6 to the five points on the scale,
the interval scale will have the numbers 7,
8,….., 11 ( instead of 1 to 5).
• The magnitude of the difference between
7 and 8 is still the same as the magnitude of
the difference between 9 and 10. It has an
arbitrary origin.
36
Ratio Scale
• Ratio scale: overcomes the disadvantage of
the arbitrary origin point of the interval
scale, in that it has an absolute (in contrast
to an arbitrary) zero point, which is a
meaningful measurement point.
• What is your age?
26
Ratio Scale
38
Ratio Scale
• The ratio scale is the most powerful of the
four scales because it has a unique zero
origin ( not an arbitrary origin).
• The differences between scales are
summarized in the next Figure.
39
The differences between scales
40
Properties of the Four Scales
Validity of the Study
• Can you trust the conclusions of the study?
• Internal Validity: The extent to which the outcomes of
the study result from the variables manipulated, measured
or selected rather than from other variables not
systematically managed.
– Instrumentation--If measuring instruments are
not reliable or valid, then their scores could be
inaccurate.
• External Validity: the extent to which the findings of a
particular study can be generalized to people and/or
situations other than those observed in the study.
Goodness of Measures
• It is important to make sure that the
instrument that we develop to measure a
particular concept is accurately measuring
the variable, and we are actually measured
the concept that we set out to measure.
43
Goodness of Measures
Goodness of Measures
• We need to assess the goodness of the
measures developed. That is, we need to be
reasonably sure that the instruments we use
in our research do indeed measure the
variables they are supposed to, and that
they measure them accurately.
45
Reliability
• Reliability of measure indicates extent to
which it is without bias and hence ensures
consistent measurement across time
(stability) and across the various items in
the instrument (internal consistency).
– If administered the same questionnaire today
and next month, would the results be
consistent?
– Are all questions related to a concept strongly
correlated?
66
More generally….
• If you take the same measurement
repeatedly will you get the same result? (is
it reliable?)
– If you step on the scales repeatedly, do you get
the same number?
• Is that result accurate? (is it valid?)
– Is that number accurate? (are your scales 5
pounds lighter than at the doctor’s office?, do
they give a result of 150 no matter what is
weighed?)
Validity
• Validity tests show how well an instrument that is
developed measures the particular concept it is
intended to measure. Validity is concerned with
whether we measure the right concept and is it
accurate.
• Several types of validity tests are used to test the
goodness of measures: content validity,
criterion-related validity, and construct
validity.
48
Content Validity
• Content validity ensures that the measure
includes an adequate and representative set of
items that tap the concept.
• The more the scale items represent the domain of
the concept being measured, the greater the
content validity.
• In other words, content validity is a function of
how well the dimensions and elements of a
concept have been delineated.
49
Face validity
• Does the test appear to test what it aims to
test?
– Experts
– Participants
– Researchers
Criterion-Related Validity
• Criterion-Related Validity is established when
the measure differentiates individuals on a
criterion it is expected to predict. This can be done
by establishing what is called concurrent validity
or predictive validity.
• Concurrent validity is established when the scale
discriminates individuals who are known to be
different; that is, they should score differently on
the instrument as in the following example.
51
Criterion-Related Validity Example
• If a measure of work ethic is developed and
administered to a group of welfare recipients, the
scale should differentiate those who are
enthusiastic about accepting a job and glad of a
opportunity to be off welfare, from those who
would not want to work even when offered a job.
52
(Cont.)
• Those with high work ethic values would not
want to be on welfare and would ask for
employment. Those who are low on work ethic
values, might exploit the opportunity to survive on
welfare for as long as possible.
• If both types of individuals have the same score
on the work ethic scale, then the test would not be
a measure of work ethic, but of something else.
53
Construct Validity
• Construct Validity testifies to how well the results
obtained from the use of the measure fit the theories
around which the test is designed. This is assessed through
convergent and discriminant validity.
• Convergent validity is established when the scores
obtained with two different instruments measuring the
same concept are highly correlated.
• Discriminant validity is established when, based on
theory, two variables are predicted to be uncorrelated, and
the scores obtained by measuring them are indeed
empirically found to be so.
54
Goodness of Measures
• Goodness of Measures is established through the
different kinds of validity and reliability.
• The results of any research can only be as good as
the measures that tap the concepts in the
theoretical framework.
• Table 7.2 summarizes the kinds of validity
discussed in the lecture.
55
Validity
.
56