MSc Applied Psychology
PYM403 Research
Methods
Validity and Reliability in Research
Validity and Reliability in
Research
-Aims of session are to consider



Internal / external validity
Threats to internal validity
Considerations and evaluation of external validity


Effective research design and implementation
Critiquing a journal article
Validity and Reliability:
Indicative Reading
Books on conducting Research Methods, e.g.,
Breakwell, G. M., Hammond, S., & Fife-Shaw, C. (1999). Research
Methods in Psychology (2nd ed). London: Sage.
Internet sites, e.g.,
http://www.socialresearchmethods.net/kb/contents.php
- Validity
http://www.socialresearchmethods.net/kb/intval.php
- Reliability
http://www.socialresearchmethods.net/kb/reltypes.php
Internal / External Validity
Internal Validity
-Extent to which we can conclude a causal relationship
between two variables
External Validity
- Extent to which we can generalise from sample, settings,
variables manipulated, and variables measured
Research as tension between Internal / External validity
Internal Validity
- “True” experiment better able to conclude IV has effect on DV
- Features of true experiment include manipulation of IVs, randomisation
techniques, control of other variables
- “Correlation does not imply causation”
- What about other quantitative designs looking for “differences between
groups/conditions” and hoping for causal explanation?
- Various threats to internal validity when design not true experiment can mean rival hypotheses - may or may not be plausible
- Your job as researcher to spot them and discuss
- Despite limitations, still very important to carry out studies “in the field”
Threats to Internal Validity
(Single Group)
-History – events between 1st and 2nd measurement
-Maturation – participant changes over time per se (i.e. not
events) e.g., older, more hungry, less motivated
-Testing – effect of taking test once on taking the scores a
second time
-Instrumentation – changes in instrument or observers /
scorers over time
-Statistical regression – particularly when groups selected on
basis of extreme scores (selection bias)
Threats to Internal Validity
(Multiple Groups)
- Selection bias – differential selection of respondents for
comparison groups (self-selected groups can be problem)
- Sources of bias interact
e.g., Selection history threat
◦Selection maturation threat
◦Selection testing threat
◦Selection instrumentation threat
Selection regression threat
- Mortality – differential loss of participants from groups
Threats to Internal Validity
(Social Interaction Threats)
- Diffusion of Treatment – “second hand training”
- Compensatory Equalization of Treatment
- Compensatory Rivalry
- Demoralization Effects
- Local History – difference in conditions in which groups
were tested
External Validity
- Research would ideally maximise both Int & Ext - not
always possible.
- In general, true experiment will maximise internal validity,
probably at expense of external validity, while field study or
observational research will gain in external validity, losing
some internal validity.
-Still very important to carry out real-life empirical research
– just need to be aware of any limitations in interpretation
External Validity: Considerations
- Sample Modelling – you first identify the population you
want to generalise to and then draw a sample from that
population
- Proximal Similarity (Gradient of Similarity) – identify
factors that are more or less similar to the original study
- Threats to External Validity
Unusual people
Unusual places
Unusual times
Proximal Similarity
External Validity: Evaluation
- Population Selection - generalizability?
- Operational Definitions - IVs / DVs operationally defined?
- Parameter Values - Are there reference groups / norms?
- Pretest - Could a pretest have influenced performance?
- Demand Characteristics
Hawthorne Effect - were groups made to feel special?
Pygmalian Effect - Were subtle cues given?
Reliability
-Inter-Rater or Inter-Observer Reliability
Used to assess the degree to which different raters/observers
give consistent estimates of the same phenomenon.
-Test-Retest Reliability
Used to assess the consistency of a measure over time.
-Parallel-Forms Reliability
Used to assess the consistency of the results of two tests
constructed in the same way from the same content domain.
-Internal Consistency Reliability
Used to assess the consistency of results across items within a
test.
Orienting Questions
Try to identify the design of the main study.
Is it a “true experiment” or not?
What are the variables, and are they manipulated?
What threats to validity? Do they try to address these?
What other questions addressed or other statistics used?
What are the good points? Any criticisms?
Critiquing Research
-Purpose is critical evaluation
1. Need to read with understanding
2. Analyse content
3. Evaluate
- Should be constructively critical (not simply negative)
- Appreciate contributions to psychology
- Consider methodology and ingenuity of design
-Should be objective
- Use third person, avoid personal opinions
Critiquing Research: Structure
- Begin by summarising research (very briefly)
- background, methodology, findings, conlcusions
- Then critique the work
- Then briefly summarise and make your conclusions
Critiquing Research:
Considerations
-Methodology
Internal / External Validity - threats to these?
- Appropriate design, sampling, operationalization of
variables, procedure, data collection, materials, etc.
Reliability - were the measures reliable?
- Contributions to current psychological knowledge
How does it fit history of research in this area?
What does it add to practical /theoretical aspects?
How important is the study in the wider world?
Critiquing a Journal Article
-Practice paper for critiquing
Rosenthal, R. & Fode, K. L. (1963). The effect of
experimenter expectation on the performance of the Albino
Rat. Behavioral Science, 8 (3), 183-189.