Uploaded by boink boinko

PSY290- Intro to Research Methods: Notes

PART I: Introduction to Scientific Reasoning
Chapter 1: Psychology is a Way of Thinking
Empiricism- basing one’s conclusions on systematic observations
Either a PRODUCER or CONSUMER psych student
○ Producer- produce research studies
○ Consumers- consume info they later apply to their work
How psychologists approach their work
○ 1. Act as empiricists and observe the world
■ Use evidence from senses and instruments to make conclusions
■ Cupboard theory: proven wrong- mother valuable bc she is source of food
vs Harlow’s comfort theory
○ 2. Test theories and revise their theories based on data
■ Theory: set of statements that describes general principles about how
variables relate to one another
■ Hypothesis: specific outcome the researcher expects to observe in a
study if theory is accurate
○ Data: set of observations
■ Theory-data cycle: systematic steps to solve a problem
■ Ex:
● Cupboard theory: Mother is valuable because source of food.
● Harry Harlow- contact comfort theory: mother is valuable because
of comfort of cozy touch
■ Bult 2 monkey mothers: one with food and one with only comfort
○ 3. Take an empirical approach to applied research and basic research
○ 4. Test why an effect worse
○ 5. Make work public
Good Scientific Theories:
- Supported by data
- Falsifiable
- Exhibit parsimony (simplicity)
- Theories don’t prove anything: data is “consistent with” a theory
Types of Research
- Applied research: done with a practical problem in mind
- For ex: “An applied research study might ask, for example, if a school district’s
new method of teaching language arts is working better than the former one.”
Basic research: enhance body of knowledge
- Ex: understand structure of visual system, memory, etc
Translational research: use of lessons from basic research to develop and test
applications to treatment
- Bridge from applied to basic research
Scientists write papers and submit to a JOURNAL that is peer-reviewed
- Journal editor sends submission to 3-4 experts
- Peer-review process is anonymous
- Other scientists can comment even after publishing
Theory: set of statements that describe how variables relate to one another
Journal to Journalism
- Journalism: kind of news we hear or see in public
- Benefits and Risks of Journalism
- Benefit: public learns about it
- However, 2 things are important
- Journalists need to report on MOST IMPORTANT scientific stories
- Is it sensational or actually important?
- Journalists must ensure the data is reported accurately
- Mozart Effect: journalists might misrepresent science when
writing for a popular audience
- Called that bc media mispresented data from
Mozart sonata listeners and an intelligence test; in
reality, only one aspect of intelligence was affected
Chapter 2: Sources of Information- Why research
is Best and how to find it
- Why you should not trust your experience over research
- Experience has no COMPARISON group
- Comparison group- enables us to compare what would happen
with or without thing we are checking
- Experience is confounded
- Confounds: several possible alternative explanations for an
- To fix these, we need to carefully control variables
- Research better than experience
- Confederate: actor playing a role for an experiment
Ex: steve and the catharsis experiment (catharsis not
actually effective)
- Experience could be an exception/ research is probabilistic
Intuition: using hunches about what seems natural or attempting to think about things logically
Ways Intuition is biased
- Being swayed by good stories
- Ex: Freud thought catharsis works and “Scared Straight” prison youth
- Availability heuristic: things that pop up easily in our mind tend to guide our
- present/present bias: we fail to think about what we cannot see
- Confirmation bias: tendency to look at info that agrees with what we know
- Bias blind spot: belief that we are unlikely to fall prey to the other biases
previously described
- Only trust authority on subject if thoughts are based on authority’s research
3 kinds of sources where psych scientists publish
- Journal articles
- Audience: other psych students and scientists
- Types:
- Empirical articles: report for first time results of an empirical
research study
- Review journal articles: summary of all published studies done in
one research area
- Can use quantitative technique called meta-analysis, which
combines results of many studies and gives number that
summarizes magnitude or effect size of a relationship
- Chapters in edited books
- Generally not first place a study is reported; usually summary of a
collection of research
- Not peer-reviewed as rigorously but only experts are invited to write
- Audience: other psychologists or psych students
- Whole books
- Not common with SCIENTIFIC books; can be for general audience
Finding Scientific Sources
- PsychINFO: database for psych sources
Google Scholar
Reading the Research
- Components of Empirical Journal Article
- Abstract- concise summary, 120 words long
- Introduction- 1st- topic of study explained
- Middle- background for research
- Final- specific research goals, questions, hypotheses
- Method- How researchers conducted study
- Results- quantitative and qualitative data + figures + statistical tests
- Discussion
- Opening: research q and methods and how well results support
- Next: study’s importance
- References- sources cited
Read with a purpose
- Empirical Journal Articles
- 2 Questions
- What is the argument?
What is the evidence to support the argument?
- Chapter and Review Articles
- Read the headings to get an idea and categorize
Research in less scholarly places:
- Psych books for general audience
- To see if they are well-written and researched, check REFERENCES
- Wikis as a research source
- Not compregensive, idiosyncratic (represent preferences of contributors), could
be incorrect or vandalized
- Popular media
- Can be helpful if journalist specializes in science written, but if not, can be
oversimplified and simply WRONG
Chapter 3: Three Claims, Four Validities:
Interrogation Tools for Consumers of Research
Variable: something that varies
- must have at least 2 levels or values
Measured variable: value observed and recorded
Manipulated variable: research controls it by assigning study participants to
different levels of the variable
- Vars that cannot be manipulated: IQ or age cause you can’t randomly
Constant- only has one level in the study
Conceptual variable: abstract concept (construct)
- Must be carefully defined at the theoretical level and these definitions are called
conceptual definitions
- Operational definition of variables (operationalization): turn a concept of interest
into a variable
3 claims
Claim: argument someone is trying to make
- Types:
- Frequency- describe a particular rate or degree of a single variable
- Measure only 1 variable and variables are MEASURED not
- Validities:
- Construct: how well variables are measured
- External: how well sample represents population
- Statistical: margin of error
- Association- argues one level of a variable is likely to be associated with a
particular level of another variable
- 3 types: positive association, negative association, and zero
- Validities:
- Construct: measure construct validity of each variable
- External: can it generalize to other populations or contexts
- Statistical: significance and strength
- Causal: one variable is responsible for changing the other
- Association → causality
- 3 criteria
- Variables are correlated
- covariance
- Causal variable came first and outcome later
- Temporal precendence: one variable comes
first in time (independent variable)
- No other explanations exist for relationship
third-variable criterion
Random assignment can help do
this to control for alternative
- Only an experiment can enable researchers to support a
causal claim
- Validities:
- Construct
- External
- Statistical
- internal
Only research not experiences can create a causal claim
Validity: appropriateness of a conclusion or decision
- Specify which validity we are talking about when we say a claim is “valid”
- Types:
- Construct validity: how well a conceptual variable is operationalized
- Basically how well a study measured or manipulated a variable
- To ensure this is met: establish that each variable has been
measured reliably
- External validity: how well the results of a study represent people and
- Statistical validity: extent to which a study’s statistical conclusions are
accurate and reasonable
- Whether or not numbers suppport claim
- Errors:
- Type I- mistakenly conclude there is an association when
there really isn’t
- “False positive”
- Type II- conclude there is no association when there really
- “miss”
Part II- Research Foundations for Any Claim
Chapter 4: Ethical Guidelines for Psychology
Historical Examples
Tuskegee Syphilis Study
- A lot of Black men had syphilis and researchers wanted to study the effect
of untreated syphilis on men’s health over time
- At the time, no treatment was reasonable because methods were risky
- Lasted 40 years and men were recruited from churches
- Were told they had “bad blood” instead of syphilis
- Dangerous tests such as spinal taps were conducted
- Prevented men from being treated even after syphilis received treatment
of penicillin
- 3 distinct categories of unethical choices
- Men were not treated respectfully becayse they were lied to with
no informed decisions
- Men were harmed bc they were not given treatment
- Researchers targeted a disadvantaged social group
Milgram Obedience studies
- Subjects thought they were punishing and torturing a learner who was
really an actor
- 65% of the participants obeyed but obedience decreased when learner
sat in the room
- Also dropped when experimenter was giving instructions from
down the hall or over phone
- Was it ethical?
- Some say it was bad cause it was stressful to the participants
- Some say it was okay because the participants were debriefed
and ensured learner wasn’t harmed
Core Ethical Principles
Nuremberg code as a result of the Nuremberg Trials of Nazis
Declaration of Helsinki- guides ethics in medical research and practice
Belmont Report:
- Commission of professionals gathered in Belmont Conference center in Elridge,
Maryland at request of Congress in response to Tuskegee
- Outlines 3 main principles for guiding ethical decision making
- Respect of persons
2 provisions:
- Individuals should be treated as autonomous agents free
to make up mind (with INFORMED CONSENT)
- Some people have less autonomy and are entitled to
special protection when it comes to informed consent
- Ex: children, people with disabilities and prisoners
- Beneficence
- Protect participants from harm
Examples of action to follow this:
- To prevent release of private info:
- Anonymous study- no potentially identifying info
- Confidential study- some info collected but it is not
- Fair balance between people who want to participate in research
and ones who benefit from it
Common Rule:
- Describes detailed ways Belmont Report should be applied
- Fed funded agencies in the US must follow this
Guidelines for Psychologists
APA outlines 5 general principles for guiding individual aspects of ethical behavior
1. Respect for persons
2. Beneficience
3. Justice
4. Fidelity and responsibility
a. No sexual relations with students or clients and avoid prior relations
5. Integrity
a. Profs obligated to teach accurately and therapists required to stay current
APA has 10 specific enforceable ethical standards
- Ethical Standard 8
8.01- IRB (Institutional Review Board): committee responsible for
interpreting ethical principles and ensuring that research using human
participants is conducted ethically
- Mandated by federal law in the US if federal money is involved
- 5 or more people
- At least one scientist
- At least one with academic interests outside sciences
- At least a community member with no tiest to institutions
- When prison discussed, one must be a prisoner advocate
8.02- Informed consent: researcher’s obligation to explain study to
potential participants in every day language and give them chance
whether or not to participate
- In some cases, APA standards indicate informed consent
procedures not necessary
- If study is not likely to cause harm
- If study takes place in an educational setting
- obtaining informed consent also involves informing people
whether the data they provide in a research study will be treated
as private and confidential
8.07- Deception
- Researchers witholding some details of the study from
participants—deception through omission
- actively lied to them—deception through commission
- Is deception ethical?
- In study, researcher must still uphold principal of respect
by informing of risks and benefits
- Benefience: ethical costs and benefits of study
- APA principles and federal guidelines require researchers
to avoid using deceptive research designs except as a last
resort and to debrief participants after the study
8.08- Debriefing
- researchers describe the nature of the deception and explain why
it was necessary and describe study design
- Nondeceptive studies sometimes have it too
- Purpose is to make participation in research an educational
Data Fabrication (Standard 8.10) and Data Falsification
- Unethical and has far-reaching consequences
- Data Fabrication (8.10)
- Researcher invent data that fits hypotheses
Data falsification
- Researchers influence a study’s results
Plaigiarism (8.11)
- defined as representing the ideas or words of others as one’s own
- To prevent this, cite all ideas that are not one’s own
- Even when paraphrasing, cite last name and year of publication
Animal Research (8.09)
- By APA, research must
- Care for animals humanely
- Use as few animals as possible
- Must be sure research is valuable enough to justify it
- Animal Welfare Act in US
- Must have IACUC (Institutional Animal Care and Use Committee) that
must approve project
- 3 or more members
- At least 1 vet
- At least 1 practicing scientist familiar with animal research
- At least 1 member of local community
- First, must submit scientific justification and then IACUC monitors
treatment of animals by inspecting labs every 6 months
- Animal Care Guidelines
- Replacement: researchers should find alternatives to animals in research
when possible
- Refinement: researchers must modify experimental procedures to
minimize distress
- Reduction: adopt designs that use the fewest animals possible
- Arguments of Animal Rights groups
- Animals r just as likely as humans to experience suffering
- Animals have inherent rights equal to those as humans
Chapter 5: Identifying Good Measurement
Ways to measure variables
- 3 types:
- Self-report
- Recording people’s answers to questions about themselves
- Observational
- Recording observable behaviors
- Such as IQ tests
- Physiological
Recording biological data such as brain activity, hormone levels,
or heart rate
Operational definition of variable: how research decides to measure or
manipulate the conceptual variable
Levels of operational variables can be coded using different scales of measurement
- Categorical variables (or nominal variables)
- Levels are categories
- Examples: sex (M and F), species
- Quantitative variables
- Coded with meaningful numbers
- Ordinal scale: numbers represent ranked order
- Interval scale
- Criteria
- Numbers represent equal intervals
- There is no true zero
- Ratio scale
- Numbers represent equal intervals AND value of 0 truly
means nothing
- How consistent the results of a measure are
- Types:
- Test-retest reliability- researcher gets consistent score every time with
- To assess: measure same set of participants at least twice within
time frame and then compute r
- Interrater reliability- consistent scores are obtained no matter who (which
researcher) measures variable
- To assess: ask 2 observers to rate same participants at same time
and compute r
- Kappa measures extent to which 2 rates place participants into the
same categories
- Internal reliability- study participant gives consistent answers no matter
how research phrases question
- Relevant for measures that use more than 1 item to get at the
same construct
- Ex: question phrased in multiple ways
- Cronbach’s alpha or coefficient alpha
- Statistic that measures whether or not scale has internal
- Collect data on scale from large sample of participants and
compute all possible correlations
Closer alpha is to 1, better reliability but 0.7 or
higher is sufficient
Using correlation coefficient r to quantify reliability
- Number r indicates how close the dots are to a line drawn through them
- Strong when dots are close to line
- R is 0 when relationship is weak and close to 1 and -1 when relationship
is strong
Validity: whether operalization is measuring what it is supposed to measure (construct
- Researchers have to evaluate measure’s validity through research
- What is weight of evidence in favor of this measure’s validity?
Ways to subjectively assess validity:
- Face validity: is it subjectively considered to be a plausible operalization of the
conceptual variable
- Content validity: a measure must capture all points of your defined construct
Empirical ways to assess validity:
- Criterion validity- whether measure under consideration is associated with a
concrete BEHAVIORAL outcome that it should be associated with
- Known-groups paradigm: researchers see whether scores on the
measure can discriminate among two or more groups whose behavior is
already confirmed
- Convergent validity- pattern of correlations with measures of theoretically similar
- Divergent validity- pattern of correlations with measures of theoretically dissimilar
- Test should NOT correlate strongly with measures of constructs that are
different than what we are testing for
- For example: if we’re testing for depression, we get a test that
thinks other diseases are depression
Relationship between reliability and validity
- Although a measure may be less valid than it is reliable, it cannot be more valid
than it is reliable
- If a measure does not even correlate with itself, then how can it be more
strongly associated with some other variable?
- Reliability is necessary (but not sufficient) for validity
- Usually will find both reliability and validity info in the Method section in a journal
Chapter 6- Surveys and Observations: Describing what people do
Construct Validity of Surveys and Polls
Survey or poll: when people are asked about their social or political opinions
Question Formats
- Open-ended questions- can provide spontaneous and rich info
- Drawback is that responses must be conded and categorized and this can
be time-consuming
- Forced-choice questions: pick best of 2 or more options
- Ex: narcissistic Personality inventory: choose 1 statement of 2 for 40 pairs
- Likert scale
- People presented with statement and asked to rate on a rating scale
(strongly agree, agree, etc)
- If diverges slightly, Likert-type scale
- Ex: Rosenberg self-esteem inventory
- Semantic differential format: slight difference where respondents are
asked to rate a target object using a numeric scale anchored with
adjectives (for ex: 5 star scale)
Writing Well-worded Questions
If people answer questions that suggest a particular viewpoint, some ppl change
Ex: leading questions and specifying groups
If researchers want to measure how much wording matters for their topic, they word
questions more than one way and test it
Double-barreled questions- asks 2 questions in 1
- Poor construct validity bc people might be responding to the first half of the
question, the second half, or both
Negatively worded questions
- Can cause confusion, reducing construct validity
- In case of this, researchers can ask 2 ways and use Cronbach’s alpha to see if
people respond similarly
Question order
- Way to control for this is to prepare different versions of a survey with the
questions in diff sequences
- Or if the question we are talking about is the first question
Encouraging Accurate Responses
Self-reports are often ideal, cheap, provide meaningful info
- Might also be the only option
Response sets (or nondifferentiation)- short cut respondents take when answering
survey questions
- Ppl might develop a consistent way of answering all the questions, especially to
the end of a long questionnaire
- Weaken construct validity
- One response- acquiescene (yea-saying): people say yes to every item
- Instead of measuring accurately, survey could be measuring tendency to
agree or lack of motivation to think clearly
- How to look for this?
- Include reverse-worded items
- Drawback is that sometimes it creates negatively worded
- Fence sitting- playing it safe by answering in the middle of a scale
- Can be if Q is controversial or unvlear
- Way to solve for this
- Take away the neutral option
- Drawback: if people really do not have an opinion,
choosing a side is an invalid representation
- Use forced-choice questions
- D: can frustrate ppl who feel their opinion is in middle of
- Could make thing where IDK is written if person
volunteers ambivalence
Socially desired responding (or faking good)- if ppl embarrassed of unpopular opinion,
they will not tell truth
- Another less common- faking bad
- To prevent, research tells respondents that their responses are anonymous
- Not perfect solution bc anonymous respondents may treat surveys less
- Another way- researchers include special survey items that target socially
desirable responders
- Flagged if they agree with too many of those items
- Another way: researchers ask friends to at them
- Another: using computerized measures to evaluate people’s implicit opinions
about sensitive topics
- Ex: implicit association test
Self-reporting more than they can know
People cna untintentionally inaccurate response even if they actively volunteer a
People’s justifications might be unintentionally misguided
Self-reporting memories of events
- Ppl’s memories are not very accurate about events they participated in
- For ex: flashbulb memories about activities during dramatic events
- To test, administer test day after event and then again a few years
- Findings
- Overall accuracy is very low
- People’s confidence in memories’ accuracy is unrelated to
their true accuracy- no diff
Rating Products
- One study: little correspondence between amazon 5 starts and ratings by
Constumer reports, a product rating firm
- Customer ratings correlated with cost of product and prestige of brand
Construct Validity of Behavioral Observations
Observational research: researcher systematically observes and records
- Can be basis for frequency claims
- Can be used to operationalize variables in association and causal claims
Observing how much ppl talk (Melh et al.)
- Device given measures sound at 12.5 min intervals and records
everything person says
- Published data that showed women and men showed same level of
Observing parent behavior at a hockey game to batle stereotype about parent
fights at youth hockey
- False- mostly positive comments
Observing families in the evening
- Video crews follow parents and assistants coded behaviors
- Rating from cold to neutral to happy
Making reliable and valid observations
Construct validity threatend by 3 problems
Observer bias- observers’ expectations influence interpretations
Observer effects- observers change behavior of those they are observing; participant
changes behavior to meet expectations
- Ex: maze-bright and maze-dull rats
- Ex: Clever Hans horse- horse very good at detecting questioner head
To prevent observer bias and effects:
- Clear rating instructions called “codebooks”- precise statements of how variables
are operationalized
- Using multiple observers- allows measure of interrater reliability
- Still not perfect- observers might share same bias
- Other methods
- Masled research design (blind design)- observers are unaware of purpose
and conditions of study and participants
Reactivity: change in behavior if participants know someone is watching
- Happens even with animals
- To fix- solutions
- Sol. 1- blend in
- Make unobtrusive observations
- For ex: one way mirror, act like person in crowd
- Sol. 2- wait it out
- Participant eventually forgets he/she is being watched
- Sol. 3- measure the behavior’s results
- Don’t observe behavior directly, just what it leaves behinds
Observing People Ethically
It’s ok, ppl say, in locations where ppl are aware they are in public
Secret methods- ethical in some conditions
- Ppl must obtain permission in advance or if hidden, must explain procedure at
conclusipn of study (must erase if ppl object)
Chapter 7- Sampling: Estimating the Frequency of Behaviors and Beliefs
Generalizability- does the sample represent the population?
External validity- can the study be generalized to larger populations? Is important for
frequency claims
- Of sample: is sample is adequate to entire population
- “Generalizes to” or “is representative of”
Sample types
- Biased or unrepresentative sample
- Some members of population have a much higher chance of being
- 2 ways to obtain
- Only those who they can contact conveniently
- For ex: ppl who respond to online surveys or polls
- Only those who volunteer to respond
- Self-selection
- Unbiased or representative sample
- All members have equal chance
Obtaining a Representative Sample: Probability Sampling Techniques
- Probability or random sampling- each member has equal chance of being
- Types- Simple random sampling
- Cluster sampling: option when people are already divided into
arbitrary groups
- Clusters of participants randomly selected and all
individuals in a selected cluster are used
- Multistage sampling
- 2 random samples are selected: a random sample of
clusters, and then a random sample of people within those
- Technique- stratified random sampling
- Researcher purposefully selects particular
demographic categories (strata) and randomly
selected individuals within each of the categories,
proportionate to their assumed membership in the
population, proportionate to membership in the
- Differs from cluster sampling bc
- More meaningful categories
- Sample sizes reflect proportion in
- Variation- oversampling
- Researcher intentionally overrepresents one
or more groups
- Later, final results are adjusted so that the
weights correspond to actual proportion in
Nonprobability sampling- results in a biased sample
- Convenience sampling
- Purposive sampling
- Researchers want to study only certain kinds of people, they
recruit only those particular participants
- When done in NONRANDOM WAY, called purposive
- For ex: testing effectiveness of drug to quit smoking would
choose smokers
- Snowball sampling- can help find rare individuals
- Participants asked to recommend acquaintances
- Quota sampling
- Target numbers are set for subsets of population of interest
- Diff from stratified random bc the participants are selected
Random sampling vs random assignment
Random sampling
- Enhances external validity
Random assignment
- Researchers assign participants into groups at random
- Enhances internal validity helping ensure that the comparison group and the
treatment group have the same kinds of people in them, thereby controlling for
alternative explanations.
Interrogating External Validity: What matters most?
Sometimes, unknown external validity is OK
In FREQUENCY claims, external validity is a PRIORITY
When external validity is a lower priority
- Bc random assignment prioritized over random sampling
- Sometimes sample bias isn’t relevant to the claim
Larger samples are not more representatives
If rare phenomenon, we need a large sample to locate enough instances for valid
statistical analysis
But otherwise, external validity is bout HOW not HOW MANY
Only 1000-2000 ppl really needed for public opinion
Chapter 8- Bivariate Correlational Research
Introducing Bivariate Correlations
Bivariate association = involves exactly 2 variables
3 types of associations: positive, negative, and zero
Authors present bivariate correlations between diff pairs of variables separately
Review: Describing Associations between 2 quantitative variables
Positive r = positive relationship
Describing associations with Categorical Data
When at least one of the variables is categorical, researchers use diff stats test
- More common to test whether diff between group averages is statistically
significant using a t test
A study with all measured variables is correlational
supported by a particular kind of statistic or a particular kind of graph; it is supported by a study
design—correlational research—in which all the variables are measured
Interrogating Association Claims
Important validities- construct and statistical
Construct- how well was each variable measured? Does it have good reliability and is it
measuring the intended this?
- What is the evidence for its face validity, its concurrent validity, its discriminant and
convergent validity?
Statistical- how well do data support conclusion?
- Question 1- what is the effect size?
- Larger effect size = r closer to 1 = more accurate predictions
- Errors estimating get larger when associations are weaker
- Larger effect sizes are usually more important than small ones
- Question 2- is it statistically significant?- likelihood of getting a correlation of that size just
by chance
- P < 0.05
- Small effect size will be statistically significant if identified in a very large sample
- Question 3- could outliers be affecting association
- In bivariate- outliers mainly problematic when they involve extreme scores on
both variables
- Matter most in small sample
- Question 4- Is there restriction of range?
- If there is not a full range of scores on one variable, it can make correlation
appear smaller than it is
- Correction for restriction of range formula or recruit more ppl
- Can apply if one variable for some reason has little variance
- Questio 5- is the association curvilinear?
- Could fail the r test cause the r test measures linear
Internal Validity- can we make a causal inference from an association?
we must guard against the powerful temptation to make a causal inference from any association
claim we read
Must satisfy 3 causal criteria
- Covariance
- Temporal precedence- directionality problem
- Internal validity- third-variable problem
When we think of a reasonable third variable explanation for an association claim, how do we
know if it’s an internal validity problem?
- We check if correlation is present in subgroups
spurious association- the bivariate correlation is there, but only because of some third
External Validity- To Whom Can the Association be generalized?
Size does not matter as much as the way sample was selected from a population of interest
- If not random sample, can’t be sure results will generalize to the population
How important is external validity?
- U can accept study results and leave generalization question to next study
Moderator- when the relationship between two variables changes depending on the level of
another variable, that other variable is called a moderator
In correlational research, moderators can inform external validity.
- When an association is moderated by residential mobility, type of relationship, day of the
week, or some other variable, we know it does not generalize from one of these
situations to the others.
Chapter 9- Multivariate Correlational Research
Reviewing the 3 causal criteria
Temporal precedence
Internal validity
Establishing temporal precedence with longitudinal designs
A longitudinal design can provide evidence for temporal precedence by measuring the same
variables in the same people at several points in time.
Interpreting results:
- Multivariate design gives 3 types of correlations
- Cross-sectional- are 2 variables, measured at same point in time, correlated?
- Auto- evaluate associations of each variable with itself across time
- Cross-lag- show whether the earlier measure of one variable is associated with
the later measure of the other variable
- Help establish temporal precedence by addressing directionality problem
- 3 results:
- One correlation is significant
- Opposite correlation is significant
- Both correlationships are significant
Ruling out third variables with multiple-regression analyses
Basically means to measure other potential interfering correlations
By conducting a multivariate design, researchers can evaluate whether a relationship between
two key variables still holds when they control for another variable
- If researchers take the relationship with third variable into account, is there still a
correlation with our original variables?
- Similar to looking within subgroups
Criterion variable (dependent)- the one researchers want to predict
Predictor or independent variables
Using betas to test for third variables
- One beta for each predictor variable
- A positive beta, like a positive r, indicates a positive relationship between that predictor
variable and the criterion variable, when the other predictor variables are statistically
controlled for.
- Can’t describe effect size like in r- betas change depending on predictor variables being
controlled for
Coefficient b- The coefficient b represents an unstandardized coefficient.
A b is similar to beta in that the sign of b—positive or negative—denotes a positive or
negative association (when the other predictors are controlled for). But unlike two betas,
we cannot compare two b values within the same table to each other.
- The reason is that b values are computed from the original measurements of the
predictor variables (such as dollars, centi- meters, or inches), whereas betas are
computed from predictor variables that have been changed to standardized units.
P-value indicates if beta is statistically significant
Adding several predictors to a regression analysis can help answer two kinds of questions.
- First, it helps control for several third variables at once.
- Second, by looking at the betas for all the other predictor variables, we can get a sense
of which factors most strongly predict the chance of pregnancy.
Phrases for regression
- “Controlled for”
- “Taking into account”
- “Adjusting for”
Regression does NOT establish causation
- Not able to establish temporal precedence
- But can eliminate internal validity
Getting at Causality with Pattern and Parsimony
Researchers can investigate causality by using a variety of correlational studies that all point in a
single, causal direction.
This approach can be called “pattern and parsimony” because there’s a pattern of
results best explained by a single, parsimonious causal theory
- Parsimony- degree to which theory provides simplest explanation, making fewest
By converging studies, you can try to find the simplest correlation
- Mult studies diversity makes it hard to raise third variable explanations
In popular media
- When journalists write about science, they do not always fairly represent pattern and
parsimony in research. Instead, they may report only the results of the latest study.
- Selective presentation of scientific process
Mediator- variable that explains why of patterns between variables
Mediator vs third-variable
- when researchers propose a mediator, they are interested in isolating which aspect of the
presumed causal variable is responsible for that relationship.
A mediator variable is internal to the causal variable and often of direct interest to
the researchers, rather than a nuisance.
Mediator vs moderators
- Mediators ask: Why?
- Moderators ask: Who is most vulnerable? For whom is the association strongest?
Chapter 10- Introduction to simple experiments
Example 1: Taking notes
Scored the same on factual, long hand did better on conceptual than on laptop
Example 2: Serving Pasta
Larger bowl participants = overall ate more pasta
How do experiments establish causal claims
- Covariance
- Because every independent variable has at least two levels, true experiments are
always set up to look for covariance
- Results matter too
- Temporal precedence
- Manipulating a variable ensures it came first in time
- Internal validity
- Done by keeping controls constant
- there can be several possible alternative explanations, which are known as confounds, or
potential threats to internal validity
Threats to internal validity
- Design confound: experimenter’s mistake in designing the independent variable
- second variable that happens to vary systematically along with the intended
independent variable and therefore is an alternative explanation for the results
- Only a problem for internal validity if the problem shows systematic variability with the
independent variable
- If unsystematic (random or haphazard) variability, it is not a confound
- But still can obscure patterns
- Selection effects- when the kinds of participants in one level of the independent variable
are systematically different from those in the other
- Can happen if experimenters let participants choose group they want to be in
- Can avoid with
- random assignment
- Matched groups
- Within matched set for a particular variable, researcher randomly
assign one person to one group and the other to another
- Requires more time and resources
Independent-Groups vs Within-Groups designs
- Independent-group design (between-subjects or between-groups)
- different groups of participants are placed into different levels of the independent
- 2 basic forms of design
- Posttest-only (equivalent groups, posttest-only)
- participants are randomly assigned to independent variable
groups and are tested on the dependent variable once
- satisfy all three criteria for causation
- They allow researchers to test for covariance by
detecting differences in the dependent variable. (Having
at least two groups makes it possible to do so.) They
establish tem- poral precedence because the
independent variable comes first in time. And when they
are conducted well, they establish internal validity. When
researchers use appropriate control variables, there
should be no design confounds, and random assignment
takes care of selection effects.
- pretest/posttest
- participants are randomly assigned to at least two different
groups and are tested on the key dependent variable
twice—once before and once after exposure to the independent
- Researchers might use a pretest/posttest design if they want to
study improvement over time, or to be extra sure that two groups
were equivalent at the start—as long as the pretest does not
make the participants change their more spontaneous behavior
- Within-groups design (within-subjects)
- Only one group and each person presented with all levels of the independent
- main advantage of a within-groups design is that it ensures the
participants in the two groups will be equivalent
- treating each participant as his or her own control
- Power- probability that a study will show a statistically significant result
when an independent variable truly has an effect in the population
- Requires fewer participants
- 2 basic forms:
- Repeated-measures design
- Participants are measured on a dependent variable more than
once, after exposure to each level of the independent variable
- Concurrent-measures design
- participants are exposed to all the levels of an independent
variable at roughly the same time, and a single attitudinal or
behavioral preference is the dependent variable
- Threat to internal validity- order effects
being exposed to one condition changes how participants react to the
other condition
- Practice effects (fatigue)- participants either get better at or tired
of the task
- Carryover- some contamination carried over from ne condition to
- Use counterbalancing technique to cancel these out
- Diff. sequences
- Full- used if there are only 2 or 3 levels of indiv. var
- All possible condition orders are represented
- Partial- only some are represented
- Can use randomized order
- Or latin square to ensure that every condition appears in
each position at least once
3 disadvantage of within-groups
- Potential order effects
- But can be controlled thru counterbalancing
- Might not be possible or practical
- problem occurs when people see all levels of the independent variable
and then change the way they would normally act
- Demand characteristic- cue that leads participants to guess an
experiment’s hypothesis
Construct validity
- Dependent variables- face validity
- Independent- manipulation check and pilot studies
- A manipulation check is an extra dependent variable that researchers can insert
into an experiment to convince them that their experimen- tal manipulation
- A pilot study is a simple study, using a separate group of participants, that is
completed before (or sometimes after) conducting the study of primary interest
- may use pilot study data to confirm the effectiveness of their
manipulations before using them in a target study
- How participants were recruited- random sampling?
- Generalizing to other situations- it is sometimes necessary to consider the results of other
- If not significant, there is not a covariance
- Effect and sample size can determine covariance strength
- Indicator of standardized effect size d- measure represents how far apart two
experimental groups are on the dependent variable
- The standardized effect size, d, takes into account both the dif- ference between
means and the spread of scores within each group (the standard deviation).
- When d is larger, it usually means the independent variable caused the
dependent variable to change for more of the participants in the study.
- When d is smaller, it usually means the scores of participants in the two
experimental group overlap more
3 potential threats to internal validity
- 1. Did the experimental design ensure that there were no design confounds, or did some
other variable accidentally covary along with the intended independent variable? (Mueller
and Oppenheimer made sure people in both groups saw the same video lectures, in the
same room, and so on.)
- 2. If the experimenters used an independent-groups design, did they control for selection
effects by using random assignment or matching? (Random assignment controlled for
selection effects in the notetaking study.)
- 3. If the experimenters used a within-groups design, did they control for order effects by
counterbalancing? (Counterbalancing is not relevant in Mueller and Oppenheimer’s
design because it was an independent-groups design.)
Chapter 11- More on Experiments: Confounding and Obscuring Variables
Threats to internal validity- Design confounds: alternative explanation bc experiment was poorly designed
- Another variable varies systemically along with the intended independent
- Selection effects- different independent variables have systematically different types of
- Differences in participants and not intended independent variable caused the
- Order Effects- outcome might be caused by indepdenent vairable but also the order in
which the levels of vairables are presented
- Ppl could be getting tired, bored, or practiced
In only one-group, pretest/posttest design
Maturation effects
- Behavior that emerges spontaneously over time not because of any outside
intervention (like our intended variable)
Also note- spontaneous remission of depression in woman example CBT therapy
in book
- To prevent these, comparison group is important
History threats- something specfic happened from pre to post test
- External factor that affects most members of treatment group at same time as
- To prevent these, use comparison group
Regression threats- if when measured at time 1 it is extreme and closer to its usual
average value at Time 2
- Occur only when a group has an extreme score at pretest
- To prevent, use comparison groups + if initial extremes are observed, account for
Attrition threats: due to reduction in participant numbers; becomes problem if systematic
(if only a certain kind of participant drops out)
- To prevent, remove those participants from pretest average too
- Also check if they have extreme scores on pretest- more of attrition effect if they
Testing threats- change in participant due to the effects of taking a test more than once
- They become more practiced or bored
- To prevent, maybe abandon pretest and do posttest only design?
- Or comparison group also helps
Instrumentation threats- when measuring instrument changes over time; in observational
research, people might change standards for judging behavior
- To prevent, switch to posttest only or ensure pre and post tests are equivalent
- Could also counterbalance the tests (using some of each both times)
In any study
Obserer bias- researchers’ expectations influence interpretation of results
Demand characteristics- participants guess what study is supposed to be about and change
To avoid both!- do a double-blind study
- If not possible, a masked/blind study- participants know which group they are in, but not
Placebo effects- ppl who think they are receiving a valid treatment improve
- TO AVOID: include a group with placebo, one w no therapy, and one w drug
Combined Threats:
Selection-history threat- outside event affects only those at one level of independent variable
Selection-attrition threat- only one of experimental groups experiences attrition
Null effect: no significant covariance between indepedent and dependent variable
Possible reasons for it
- 1. Maybe indepedent variable rlly doesn’t affect the dependent v
- 2. Study not designed correctly
2 factors:
- Not enough between-groups diff (not enough independent var level)
- Weak manipulation- how did researchers operationalize
independent variable? (construct validity)
- Insensitive measures: researchers did not use an
operationalization of dependent variable w enough sensitivity
- So use detailed instruments
- Ceiling and floor effects
- Ceiling effects: all scores squeezed together at high end
- Floor effect: all scores cluster at low end
- Do a MANIPULATION CHECK to detect al of these
- This is a separate dependent var that experimenters
- Or design confounds could undo indepedent var effect
- Too much within-groups variability (confounding variables bc too much
stuff to think about)
- For ex: too much unsystematic variability: NOISE (error variance
or unsystematic variance)
- Makes the effect size smaller and can obscure a group
- Can be caused by
- Measurement error: human/instrument factor
that alters a person’s true score
- More random error = variability
- Solutions:
- Use reliable and precise tools
- Measure more instances- so all
possible errors can cancel each
other out
- Due to individual differences between people
- Solutions:
- Change the design to use
within-group design instead
(where you mentioned
before/after of 1 person)
- Add more participants- u get a
larger t-value and it’s easier to
find a significant t
- Situation noise- due to an external distraction
ppl might react differently
- Solution:
- Carefully control surroundings
- All these solutions increase power (likelihood a study will
return a statistically significant result when the
independent variable rlly has an effect)
- To increase power:
- Use within-groups design
- Strong manipulation
- Larger number of participants
- Less siuation noise
More power= detect more true patterns
Chapter 12- Experiments with more than 1 variable
Experiments with 2 independent variables can show interactions
- Interaction effect: whether effect of original independent variable depends on level of
other independent variable
- Interaction = difference in differences
Intuitive Interactions
- Crossover interaction: lines cross each other- “it depends”
- There is a difference in differences
- Ice cream vs pancake temperature
- Spreading interaction: not parallel and do not cross each other- “only when”
- Dog bribed and told to set
- No difference when condition not met but large difference when it is
Factorial design- used to test for interactions
- One in which there r 2 or more independent variables (factors)
- Test for each combination of independent variables
- Participant variable- variable whose levels are selected or measured not manipulated
- Example: Age
- Form of external validity- tests whether effect of independent variable generalizes across
other variables
- Testing for moderators (variable that changes relationship between 2 other variables)
Interpreting factorial results- main effects and interactions
- Main effect- overall effect of 1 independent variable on the dependent variable averaging
over levels of other independent variable (simple difference)
- 2 main effects w 2 independent vars
- Marginal means are the arithmetic means for each level of an independent variable,
averaging over levels of the other independent variable
- Simple average
- Main effects may or may not be stat significant
- However, interaction is the most important
- Think of a main effect instead as an overall effect—the overall effect of one
independent variable at a time.
- Interaction effect = difference in differences
- To describe
- Either explain both patterns
- Or use key phrases like “it depends” or “especially in”
Factorial Variations
Independent-Groups (between subjects)
both independent variables are studied as independent-groups.
- There- fore, if the design is a 2 × 2, there are four different groups of participants
in the experiment.
Within groups (repeated measures)
- both independent variables are manipulated as within-groups.
If the design is 2 × 2, there is only one group of participants, but they participate
in all four combinations, or cells, of the design.
- Requires fewer participants
Mixed- one independent variable is manipulated as independent-groups and the other is
manipulated as within-groups
Increasing # of levels of independent variable
- _ # x _ # (each is # of levels for each independent var)
- Quantity of numbers equals number of independent vars
Increasing # of independent vars = 3 vars = _ x _ x _
- 3 main effects, 3 two-way interactions and 1 three-way interactio
Identifying factorial designs in reading
- Empirical study
- Method section= study design
- Results section= are main effects and interactions significant
- Popular media
- Talk of “it depends”
- Participant variables like age, personality, gender, or ethinicity
Chapter 13- Quasi-Experiments and Small-N Designs
Quasi-experiment- researchers do not have full experimental control; researchers study
participants exposed to levels of an independent variable but they might not be able to randomly
Examples of Independent-Groups Quasi-Experiments- where there are different participants at
each level of the independent variable (nonequivalent control group design)
- At least 1 treatment + control group, but no random assignment
- Ex: ppl walking by a church building
- Researchers couldn’t choose random ppl to walk by the building
- No random assignment
Examples of Repeated-Measures Quasi-Experiment- same participants experience all levels of
an independent variable
- Researcher relies on already scheduled or planned policy that manipulates the
independent variable
- Ex: judicial decision-making
- Example of interrupted time-design- a quasi-experimental study that measures
participants repeatedly on a dependent variable (in this example, parole decision
making) before, during, and after the “interruption” caused by some event (a food
- Ex: health insurance laws- The independent variables were not controlled by
researchers—people were not randomly assigned to states that had health
insurance laws and those that did not.
- nonequivalent control group interrupted time-series design- it combines two
of the previous designs (the nonequivalent control group design and the
interrupted time- series design)
Internal Validity in Quasi-Experiments
- Main concern is internal validity
- Threats
- Selection effects- relevant only for independent-groups not
- Participants at one level are systematically different from others
- To prevent:
- can try to make matched groups to control for selection
- Or use wait-list design- all participants plan to receive
treatment but are randomly assigned to do so at different
times (like different time slot for plastic surgery)
- Design Confounds: outside variable that systematically varies with
targeted independent variable
- To prevent, can track data to analyze for design confounds
- Maturation threat- observed change emerges spontaneously over time
- To prevent: make a comparison group
- History threat- external, historical event happens for everyone in study at
same time as the treatment
- To prevent: use comparison group however, vulnerable to
selection-history effect
- Selection-history effect- historical event affects only ppl in
treatment group/comp group
- Regression to the mean- extreme mean caused by combo of random
factors unlikely to happen again
- Mainly a problem whena group is selected for differing high or
low scores at pretest
Attrition threat- people drop out of study over time and this is systematic
- Easy to check for
Testing and Instrumentation threats- testing : order effect bc participants change
- Instrumentation test: instrument could change or different test
- Comparison group can rule these out
Observer bias, demand characteristics and placebo effects: all related to
human objectivity
- All easy to interrogate
- For observer bias, you simply ask who measured the behaviors.
Was the design blind (masked) or double-blind? For
experimental demand, you can think about whether the
participants were able to detect the study’s goals and respond
accordingly. For placebo effects, you can ask whether the design
of a study included a comparison group that received an inert, or
placebo, treatment.
Balancing priorities in Quasi-experiments
- Real-world opportunities
- External validity- likelihood patterns will generalize
- Ethics
- Construct validity and statistical validity
- Excellent construct v generally and for stats, just analyze significance and effect
Are they the same as correlational studies?
- Seem similar when a quasi-experiment uses an independent-groups design— that is,
when it compares two groups without using random assignment
- Difference: researchers do more meddling in quasi-experiments
- Try harder to obtain groups by targeting key locations
Small-N designs: studying only a few individuals
- How sample selected is more important than size, for external validity
Balancing priorities in case study research- how convinced can we be?
- Experimental control: can make conclusions with good research design
- Helps study special cases
Disadvantages of small-N studies
- Internal validity- hard to pin down actual problem and cause
- External validity- may not represent general population well
- To explore generalizability- might be smart to triangulate (compae case study
results to research using other methods)
3 Small-N designs- used frequently in behavior analysis
Stable-baseline- researcher observes behavior for a baseline period before beginning treatment
- If behavior during the baseline is stable, the researcher is more cer- tain of the
treatment’s effectiveness
- If the researchers had collected a single baseline measure before the new
treatment and a single test afterward (a before-and-after comparison), the
improve- ment could be explained by any number of factors, such as maturation
(spontaneous change), regression to the mean, or a history effect. (In fact, a
single baseline record followed by a single posttreatment measure would have
been a small-N version of the “really bad experiment”; see Chapter 11.) Instead,
the researchers recorded an extended, stable baseline, which made it unlikely
that some sudden, spontaneous change just happened to occur right at the time
the new therapy began. Further- more, the stable baseline meant there was not a
single, extreme low point from which improvement would almost definitely occur
(a regression effect). Performance began low and stayed low until the
experimental technique was introduced.
- Baseline gave study INTERNAL VALIDITY
Multiple-baseline design- researchers stagger their introduction of an inter- vention across a
variety of individuals, times, or situations to rule out alternative explanations
- Helps rule out alternative explanations so INTERNVAL VALIDITY to support causality
Reversal design- a researcher observes a problem behavior both with and without treatment, but
takes the treatment away for a while (the reversal period) to see whether the problem behavior
returns (reverses)
- Reversal designs are appropriate mainly for situations in which a treatment may not
cause lasting change.
- Also, there are some ethical issues here
Evaluating 4 validities in small-N designs
- Internal validity, if conducted with good design, is quite high
- External validity- question of generalizability
- Enhanced by triangulating (combining with results of other small-N studies),
specifying population they want to generalize and sometimes, they don’t even
care about generalizing
- Construct validity- measurements reliable and valid
- Use multiple observers and checking for interrater reliability in case of
- Statistical validity- generally not traditional stats
- But effect sizes are important
Chapter 14- Replication, Generalization and the Real World
Must be replicable results
3 types of relocation studies
- Direct replication- researchers repeat original study as closely as they can
- Can never usually follow exact detail however
- Problem: if there are threats to internal validity or construct validity in original
study, they are now repeated and also doesn’t test in a new context
- Conceptual replication- researchers explore same research question but with different
- Same conceptual variables but diff procedures to operationalize
- Replication-plus-extension- researchers replicate original experiment and add variables
to test additional questions
- New tests or conditions or introduce new independent variable
Replication debate in psychology- replication crisis bc direct replication dates are bad
- Why do replication studies fail?
- One reason: contexts were manipulated to be too different
- Another: OSC only conducted one replication attempt per original study
- Other: problem with original studies
- Sample size too small or questionable research practices were employed
- HARK- hypothesizing after the results are known
- P-hacking- running diff analysis or individuals that supports conclusion
- Improcements to scientific practice
- Require larger sample sizes
- Open science- practice of sharing data openly for replication and
Preregistration of methods in advance of data collection
Scientifi literature- series of related studies conducted by various researchers that have tested
similar variables
- Meta-analysis: way of mathematically averaging the results of all published and
unpublished studies that have tested the same variables
- Strengths
- Can sort studies into moderators and compute separate effect sizes
- Usually uses all peer-reviewed data
- Can tell overall effect size
- Limitations
- File drawe problem- idea that a meta-analysis might be overesti- mating
the true size of an effect because null effects, or even opposite effects,
have not been included in the collection process
- To combat, researchers should contact colleagues for published
and unpublished data
Reliability, importance and popular media
- Good journalists give sense of literature not only the recent study
- Some summarize press releases and don’t question the study
To be important, must a study have external validity?
- Direct replication studies don’t support this, but conceptual and repl+exten can
- Generalizing to other participants- to analyze this, ask how participants were obtained
- If convenience, can’t be sure
- How is important rather than sample size
- U can choose which population u represent
- Generalizing to other situations
- Conceptual replications illustrate this
- Does it generalize to real-world?- ecological validity or mundane realism
- Does it have to be generalizable to many people?
- Not rlly; depends on research mode of researcher
- Theory-testing- designing correlational or experimental research to
investigate support for a theory
- Generalization- researchers want to generalize the findings from the
sample in a previous study to a larger population
- Frequency claims are always in generalization more
- Association and causal are sometimes in generalization more
Cultural psychology: a special case of generalization mode (subdiscipline of psychology focusing
on how cultural contexts shape the way a person thinks, feels, and behaves)
- challenged researchers who work exclusively in theory-testing mode by identifying
several theories that were supported by data in one cultural context but not in any other
Muller-Lyer illusion: ppl in N America and Europe fall fo this illusion, bu not around whole
- he Segall team used their global data to conclude that people who grow up in a
“carpentered world” have more visual experience with right angles as cues for
depth perception than people who grow up in other societies.
- Researchers cannot take generalization for granted
Figure and ground
- japanese students tended to focus on the background first and focused on the
background of the fish, but N Americans recognized the fish primarily
Theory testing using weird participants
- Western, educated, industrialized, rich, democratic
Does study have to take place in a real-world setting
- Real-world (field setting)
- Built in advantage for external validity
- However, can also be replicated in lab (experimental realism)
- Generalization mode and the real world
- Because external validity is of primary importance when
researchers are in gener- alization mode, they might strive for a
representative sample of a population. But they might also try to
enhance the ecological validity of a study in order to ensure its
generalizability to nonlaboratory settings.
- Theory-tesing mode and the real world
- When a researcher is working in theory-testing mode, external
validity and real- world applicability may be lower priorities.
- Theory-testing mode prioritizes internal validity at the expense of
all other considerations, including ecological validity.