MMC 9002 (500) Researching Communication
Fall 2007 Lombard
Validity and Reliability, Sampling, Levels of Measurement
Validity and Reliability
Types of variables
• Independent variable vs. dependent variable
• Extraneous
• Control
• Uncontrolled
Relationship between variables
• Qualitative variables
•If one variable changes, other does
• Quantitative variables
•Positive or direct
•Negative or inverse
•Curvilinear
• Qualitative and quantitative
•Differences between various levels
• Causal
•Association
•Time-order
•Non-spuriousness
Ways to measure variables
• Direct
• Self-report
• Single or multiple items
• Observation
• Indirect
• Unobtrusive: collect data without individual’s knowledge
Validity
• Internal (causal): Did A really produce B?
• External: How well does the sample generalizes to a larger population?
1
• Measurement: Extent to which operation measures the concept as intended
Threats to internal validity
• History: events in real world affect people between 1st and 2nd test.
• Maturation: change due to time of task; aging, hungry, tired
• Testing: effect of being tested again
• Instrumentation: wording reactions
• Regression: effect of extreme scores moving to middle on retest
• Selection: bias in sample
• Mortality: drop out
Threats to external validity
• Reactivity: test situation different from real world
Reliability
• Deals with the operational level
• Extent to which measure obtains same result again and again
• Or the extent to which multiple measures measure the same thing
Ways to examine reliability
• Stability: consistency of instrument over time
•Test-retest
• Internal consistency: consistency of item performance
•Split-half & Cronbach’s alpha
• Equivalency: correlation between two forms of a test or different judges on the
same test
•Intercoder reliability
•Alternative forms
2
Tradeoff between validity and reliability:
Sampling
The basic idea of sampling is that we seek knowledge about an entire population
based on some cases - because it is impossible to measure everyone in a short
amount of time.
Terms:
1. Population: A group or class of subjects, variables, concepts, etc.
3


Must define precisely before you can sample
This is the conceptual definition - the group in abstract.
2. Sampling Frame: The operational definition of the population - what you will
actually choose the sample from.
 Hopefully sampling frame and population are identical.
 Differences between the two could mean systematic biases.
e.g., how good are phone books as a sampling frame?
3. Census: The case of measuring every member of a population.
4. Sample: A subset of the population, selected in any fashion. Ideally, a
sample should be representative of the population, but this is seldom
possible or even knowable.
Two types of sampling designs: Rules for selecting for sample.
1. Probability sampling: All elements have an equal chance for selection in the
study.
 This allows researchers to calculate the amount of sampling error
present in the study.
 A systematic selection procedure.
 No bias on part of investigator.
 Can apply statistical methods to the results.
a. Simple random sample: Every unit of sampling frame has an
equal chance of being selected.
 e.g., items selected by random number table.
 No cases are favored.
 Random start, then systematic sampling.
b. Stratified sample: A segment of the population is defined as
important, and a random sample is take from each level of the
segment.
 Used to ensure sufficient representation of small segments.
 This method is better, more efficient (smaller sample
needed) when the strata are directly related to the
dependent variable.
e.g., men and women as strata for drinking.
c. Cluster sample: A multistage design
 e.g., use tracts or clusters: Randomly select county, districts
within county, blocks within districts, households within
blocks, etc.
 Very cost efficient - don’t have to send people everywhere.
4
Simple random sample:
Systematic random sample:
Stratified random sample:
5
Cluster sample:
6
2. Non-probability sampling: Selection based on means other than chance.
a. Convenience: Researcher uses people most available.
 people in mall, a class of students
 used in market research, experiments
b. Purposive sample: Selection based on specific criteria
 Used in field observation
c. Quota sample: Selection based on known, predetermined
percentages that exist in real world.
Factors to consider in determining what sample size to use:
1. Project type and purpose: New, exploratory vs. fine-tuning previous research.
2. Project complexity: How will data be used, how many variables, how much
precision needed?
3. Resources: Money, time, etc.
4. Population heterogeneity: More heterogeneous requires more cases.
5. Desired precision: Larger sample provides more precision.
6. Sampling design: Some methods more efficient (e.g., stratified over S.R.S.)
Generally 400-500 is sufficient if you don’t have too complex of analyses or too
many variables.
Eliminate non-random error from:
Incomplete sampling frames
Incomplete data collection
Definitions in describing samples:
Statistic: A characteristic of a sample.
Parameter: A characteristic of a population.
Measures of central tendency:
Mode: Most frequent answer
Median: Mid point
Mean: Average
7
Measure of dispersion:
Range: Difference between highest and lowest value
Variance: A measure of how condensed or spread out the values are
around the mean.
Standard deviation: Square root of the variance.
[See example in Singleton, pages 141-3.]
Levels of measurement
Four levels at which variables can be measured
• Nominal
•Categorical, classification
•Property of equivalence
•Exhaustive and mutually exclusive
• Ordinal
•Rank along a specific dimension
•Properties of above and order
• Interval
•Properties of above and equal spacing
• Ratio
•Properties of above and a true zero point
8
9
Discrete versus continuous data:
10