Four Primary Areas of
Quantitative Research
Measurement – Construct Validity
Sampling – External Validity
Design – Internal Validity
Statistics – Statistical Conclusion Validity
Scales of Measurement
Nominal: categorical measurement. We use this when we group observations
of the same attribute together, for example, putting people into religious
categories (p. 138)
Ordinal: goes one step beyond nominal level in that we not only group like
observations together but also rank them. For example, the question “ Would
you rate your social support as very good, satisfactory, or very poor?” asks for
an ordinal judgment. Ordinal measures may use numbers for convenience in
coding answers, for example, very good (3), satisfactory (2), and very poor (1).
However, such numbers for ordinal categories may serve only as handy,
abbreviations or labels, and you could not conclude that someone answering
with a “3” had three times as much social support as someone answering
with a “3” (p318)
Interval: not only group equivalent observations together in ordered
categories, but also consider the interval between adjacent categories as
equal, but no natural zero. Such as temperature, differences make sense, but
ratios do not. You can think of the meaning of interval level measurement if
you think of it as “equal-internal” measurement (p.329)
Ratio: includes all of the characteristics of the interval level plus the
additional one of having a true zero point that allows dividing one measure by
another (p.329)
Measurement –
Primary Concepts
Reliability: refers to the degree to which observed scores are “free from
errors of measurement” (APA, 1985, p. 19). We can gauge reliability by the
consistency of scores (p. 76) – CONSISTENCY
Construct Validity: refers to the appropriateness, meaningfulness and
usefulness of the specific inferences made from the measures (APA, 1985,
p.9) (p. 76) - we use the score and how will it reflects the construct
Sampling –
Primary Concepts
Population: Collection of all elements (3rd grade teachers) to whom survey
results are to be generalized (US). (p. 348).
Sample: Subset of individuals selected from a larger population (p. 350)
Sampling Methods
Probability sampling: The actual selection of elements from the frame must
give the elements in the frame an equal probability of selection. Random
sampling provides the best way of achieving equal-probability sampling (p.
128). BETTER FOR GENERALIZATIONS; MORE RIGOROUS
Non-probability sampling: includes any method in which the elements have
unequal chances of being selected. One such method, called convenience
sampling, depends on the availability of respondents. In this procedure,
subjects select themselves (p. 129)
External Validity
Generalizability of the study’s findings to other populations, places or times
(p. 345)
Design
True Experimental Designs
(Random Assignment to Control/Comparison Group). More desirable.
Control group is randomly assigned. Random selection from population into
sample; random assignment of control/treatments groups.
Quasi-Experimental Designs
(Control/Comparison Group without Random Assignment)
Pre-Experimental Designs (No Control/Comparison Group)
Internal Validity
Truthfulness of the assertion that the observed effect is due to the
independent variables in the study (p. 346). Judge the relationship between
the independent and dependent variables. RELATED TO DESIGN
Descriptive Statistics
Describes the sample. A set of methods to describe data that we have
collected (mean, standard deviation)
Inferential Statistics
Make inferences about population. Tests probability of sample data being
drawn from the population defined by hypotheses, this is a set of methods
use to make a generalization, estimate, predication or decision.
Statistical Conclusion
Validity
Refers to inferences about whether it is reasonable to presume that a
relationship exists between variables (introduction presentation)
FERPA
Family Educational rights and privacy act (1974) is a federal law that
protects the privacy of student education records (ethics presentation)
National Research Act and
Institutional Review Board
(IRB):
Committees established by us federal regulations at each research
institution to protect human subjects from abuses through prior review of
research proposals
Risk
Exposure to the possibility of physical, psychological or social injury as a
result of the study
Coercion Anonymous
Coercing people to participate require the participants cannot be identified
Confidential
Require the confidentiality of the participants
Reliability
Refers to the degree to which observed scores are “free from errors of
measurement” (APA, 1985, p. 19). We can gauge reliability by the
consistency of scores (p. 76). Reliability is a necessary, but NOT sufficient
condition for validity, reliability can also create a tension with validity.
Reliability Coefficient
Test-retest: short time between tests, stability (error due to time)
Alternate forms: forms match better in content and stat properties,
equivalence (error due to form or internal consistency)
Inter-rater: standardizing procedure including training, rubrics and
monitoring. Scorer or rater consistency.
Split-half: reliability is higher when forms are matched in content and stat
properties, internal consistency (consistency across items – content
sampling error, flawed items)
Internal consistency: longer tests: wide range of individual variability on
construct; freedom from distractions, misunderstandings, use of items of
medium difficulty on cognitive measures (two primary methods for increasing
test reliability), consistency across items – content sampling error, flawed
items.
Standard Error of
Measurement (SEM)
Alternative index for reporting random error based on confidence intervals
(reliability presentation)
Uses and Advantages
Interpretation – 68% of the time the true score is in the interval of x +/- SEM)
95% of the time the true score is in the interval X +/- 1.96*SEM), it is sample
from estimation also important (Reliability, presentation)
Construct Validity
Definition-Construct Validity-“refers to the appropriateness, meaningfulness,
and usefulness of the specific inferences” made from the measures (p.76)
Construct Validity
(Sources of Evidence)
Content-analysis of the relationship between the test’s content and the
construct of interest. It refers to the themes, wording, and format of the
items tasks, or question on a test, as well as the procedural
Substantive /Response Process: Theoretical and empirical analyses of the
response processes of examinees are used to determine the fit between the
construct and detailed nature of the examinees actual performance or
responses.
Internal: Analysis of the internal structure of a test can indicate the degree to
which the relationship among test items and test components conform to the
construct on which the proposed test score interpretations are based
(construct validity – presentation)
Relationships with External Variables: Analysis of relationship of test scores
to variables external to the test (construct validity – presentation)
Consequences: Appraises the value implications of score interpretation as a
basis for action as well as the intended and unintended consequences of test
use, especially in regard to sources of invalidity relation to issues of bias,
fairness and distributive justice (construct validity, presentation)
Construct Irrelevant
Variance: “surplus
construct irrelevancy”
(construct validity,
presentation)
Construct irrelevant difficulty: Aspects of the task are extraneous to the focal
construct – can make the test more difficult for some individuals or groups
Construct
Underrepresentation:
The test is too narrow and fails to include important dimensions of the
construct (construct validity, presentation). For example, one test is one test
is trying to measure the mathematics curriculum including probability,
algebra, measurement, data analysis, etc. however, if the test only includes
the terms relating algebra.
Internal Validity
Definition – Isolation; Refers to whether or not the relationship between two
variables; refers to the truthfulness of the claim that one variable causes
another
Construct irrelevant easiness – when extraneous clues in the time or format
results in correct responses
Threats to Validity
History: Refers to the threat that some coincidental event outside the study
caused the observed change; can control by reducing time between measures
The situation in which some specific event occurs during the study, which in
turn affects the results. The specific event is an effect other than the
experimental treatments. To prevent this threat use a control group, shorten
time between testing, select a dependent variable less prone to history
effects, or insulate participants.
Maturation: refers to any naturally occurring process within individuals as a
function of time per se that may cause a change in their performance.
Processes include fatigue, boredom, growth, or intellectual maturation. Time
threat to internal validity in which internal or normal developmental
processes cause the observed change. To prevent this threat, conduct the
study over shorter period of time or use a control group with a comparable
maturation rate.
Testing: refers to a response to the pretest that causes the observed change
in the outcome variable; can reduce by disguising the pretest or dropping it
completely
Instrumentation: when observed changes results from shifts in the way
measures are collected; can be controlled by carefully standardizing and
monitoring the measurement procedures
Regression: Comes from the tendency of scores from unreliable measures to
move toward the mean on retest; affects one group studies that select
subjects for their extreme scores; To prevent avoid the selection on the basis
of extreme scores or create a control group of extreme scores. Can also use
highly reliable measures.
Mortality: subject attrition from pretest to posttest, which casts doubt on
validity of the study; protection of this threat cannot be provided by a control
or random assignment
Selection
Selection Interactions
Design
True Experimental Designs
Role of control/comparison groups
Detect and explain possible confounding variables
Role of random assignment
Increases isolation by removing the relationship between the
treatment and other confounding variables. Results in
equivalent groups
Experimental Designs
Designs that involve some form of control/comparison group but do
NOT use random assignments. Generally, selection and selectioninteraction are the greatest type of threats.
Pre-Experimental Designs
Experimental design without control/comparison group. Weak design,
but useful when pretests are not possible (i.e., pilot testing)
Random Assignment (Internal Validity): method of placing subjects in
different conditions so that each subject has an equal chance of being in any
group to avoid systematic subject differences between the groups (p. 349)
Random Selection (External Validity): drawing a representative group from a
population by a method that gives every member of the population an equal
chance of being drawn (p. 349)
Matching: Exact equivalence on select variables – uncertainty on all others
Statistical Adjustment: Argue theoretical model – each model leads to a
different solution (Experimental design, presentation)