Table of Contents
IDENTIFY DIFFERENT RESEARCH DESIGNS: ....................................................................... 3
PARADIGMS:........................................................................................................................ 3
PARADIGM STRUCTURE: ................................................................................................................... 3
TYPES OF PARADIGMS: ..................................................................................................................... 4
RESEARCH METHODS: ............................................................................................................ 5
RCT’S: .......................................................................................................................................... 5
EXPERIMENTS: ............................................................................................................................... 6
CASE STUDY: .................................................................................................................................. 9
CROSS SECTIONAL DESIGN: ............................................................................................................. 10
LONGITUDINAL: ............................................................................................................................ 10
COMPARATIVE: ..............................................................................................................................11
SCIENTISTS AND THEIR CHOICE OF PARADIGM: ............................................................................ 12
POPPER: ..................................................................................................................................... 12
KUHN: .........................................................................................................................................13
KAHNEMAN: .................................................................................................................................13
NELSON: ..................................................................................................................................... 14
GUBA: ........................................................................................................................................ 14
RICHIE:........................................................................................................................................ 15
EVALUATE RESEARCH QUALITY: QUANTITATIVE AND QUALITATIVE ................................. 15
QUANTITATIVE RESEARCH EVALUATION: ................................................................................... 15
QUALITATIVE RESEARCH EVALUATION: ..................................................................................... 15
LIST OF BIASES, THOROUGH EXPLANATION: ................................................................... 15
SELECTION BIAS: ................................................................................................................ 15
OMITTED VARIABLE BIAS: ...................................................................................................... 16
PUBLICATION BIAS: ............................................................................................................. 16
CONFIRMATION BIAS:........................................................................................................... 16
SAMPLING BIAS: ................................................................................................................. 17
RESPONSE BIAS: ................................................................................................................. 17
RECALL BIAS: ..................................................................................................................... 17
SURVIVORSHIP BIAS: ........................................................................................................... 17
EXPERIMENTER BIAS: ........................................................................................................... 18
ATTRITION BIAS:................................................................................................................. 18
TIME-ORDER BIAS: .............................................................................................................. 18
CROSS SECTIONAL BIAS: ....................................................................................................... 18
PROXY MEASUREMENT BIAS: ................................................................................................. 18
CULTURAL BIAS: ................................................................................................................. 19
VOLUNTEER BIAS: ............................................................................................................... 19
REPORTING BIAS:................................................................................................................ 19
INSTRUMENT BIAS:.............................................................................................................. 19
MEASUREMENT BIAS: .......................................................................................................... 20
CONFOUNDING: (THIRD VARIABLE PROBLEM) ............................................................................ 20
CAUSALITY AND CORRELATION: ..................................................................................... 21
THE SCIENTIFIC METHOD: ............................................................................................................... 21
DIFFERENT VIEWS ON KEY SUBJECTS: ............................................................................. 21
PARADIGM BELIEFS: RICHIE 2020 - WATTS 2011, POPPER 1962 - KUHN 1962 ................................... 21
THEORY AND ITS NATURE:..................................................................................................... 23
SCIENTIFIC CHANGE: ............................................................................................................ 23
SCIENTIFIC CHANGE AND THEORY: ........................................................................................... 23
POINTS OF AGREEMENT: ................................................................................................................ 24
POINTS OF DISAGREEMENT:............................................................................................................ 24
DETAILED ANALYSIS: ..................................................................................................................... 25
RANDOMIZED CONTROL TRIALS: NELSON AND WATTS OPINIONS:................................................... 26
Identify different research designs:
Paradigms:
Contains all the current theory on a specific subject.
Paradigms should be able to contain more theory, however sometimes theories or dataanalysis results appear, which cannot be supported by the paradigm. In that case, we may
need to conduct further testing and ultimately evaluate the paradigm in order to fit the new
reality.
Paradigm structure:
Epistemology:
Epistemology can be understood as the way philosophy is concerned with the nature,
scope, and limits of knowledge. It investigates questions like: what is knowledge? how is
knowledge acquired? And What are the criteria for determining whether something is
true/justified?
So, Epistemology has a focus of exploring the nature of knowledge itself, including
methods and processes by which knowledge is generated, justified, and evaluated.
Ontology:
Ontology is the branch of philosophy concerned with the nature of being, existing, and the
reality of the world. Ontology does for example look into questions like “What exists?”.
Ontology focuses on fundamentals of existence and the structure of reality. Ontology seeks
to understand the nature of entities, properties, and relationships.
Methodology:
Methodology is the set of procedures or systematic approach in which one conducts
research in order to generate knowledge within a study-field.
Methodology is the systematic approach in which researchers collect, analyse, and
interpret data. The methodology section often include considerations about research
design, data collection and analysis approaches and validity of obtained data.
Types of paradigms:
Positivism:
Is based on the belief that knowledge is derived from empirical observations and scientific
methods. Positivism emphasize that objectivity is the truth and that quantitative measures
can be used to analyse this. Positivistic researchers aim a detaching themselves from what
they study, in order to obtain true objectivity and reduce possible biases.
In positivism the epistemology is that knowledge is obtainable through these empirical
observations and research methods. Positivistic epistemology put a lot of emphasis on
objectivity and the pursuit of universal natural laws which can explain the true reality.
Positivistic ontology views the true reality as objective and different from human
perception. We believe a single truth is obtainable via quantitative data analysis.
Post positivism:
In post-positivism we acknowledge the inherent limitations of positivism but maintain the
emphasis on empirical evidence and scientific methods as the “correct” approach to obtain
knowledge. We are however questionable toward the notion of objectivity as the single
source of truth.
Because, in post-positivism we recognize subjectivity and individual interpretation as a
input to scientific research. We however do still value objectivity relatively more in the
pursuit of the truth.
We often see post-positivistic researchers using mixed methods research approaches to
collect, analyse and interpret data.
Constructivism:
In constructivism we believe knowledge is constructed by individuals based on their
experiences, interpretations, and beliefs. Ontologically, constructivism see the reality as
subjective in nature and as a social construct. We believe that multiple realities exists and
that they are more context related rather than objective fixed.
In constructivism qualitative research is the go-to approach. Especially, interviews,
observations, and participant observations are favoured by constructivist researchers as
these allow the researcher to explore the subjective reality of different participants.
Research methods:
RCT’s:
RCT: Sample participants are randomly assigned into two groups. The treatment group
(receives the treatment (independent variable effect) / more of it), and the control group
(does not receive independent variable effect / receives less of it).
All else must be equal, in order for the study to qualify as RCT. Otherwise it will be
categorized as a Quasi experiment where some requirements of RCT is fulfilled, but not all.
Internal and external validity:
Internal validity:
When the rules of RCT are followed strictly, internal validity will be established because the
effects of the independent variable will be highly isolated.
- We can still statistically test the uniformity of our two groups to see if any
significant differences exist. This is especially prevalent for smaller sample
sizes.
Internal validity relies on participants not knowing the conditions which other participants
are in. Knowledge of this could influence their behaviour.
External validity: things to consider when evaluating: RCT´s often have lower ext. valid.
Sample: Is it generalizable towards other samples / populations / countries / economics?
Setting: The overall environment of the study, tasks presented to participants, and time of
day when study is conducted. How generalizable to other situations is this environment and
similar / different tasks. Is time-of-day generalizable to other times?
History: The time of experiment (seasonality), and historical events taking place during
experiment (Example: Corona or war). Has the study been caried out during multiple
seasons or in times of non-crisis and crisis? If yes, this will enhance the generalizability of
the study (Of-course depending on the study results).
Hawthorne effect: Consider whether generalizability could be affected, simply by the fact
that participants know they are being observed? This has historically been shown to alter
the conduct, readiness, and opinions of people. - We may not obtain fully raw-cut opinions
of participants.
These criteria for evaluating external validity of a study should not be used in a yes/no or
established / not established fashion, but rather create a framework for discussing possible
generalizable scenarios, and situations which the study likely cannot be generalized
towards.
Important to note: External validity build upon established internal validity. Thus, if internal
validity is non-existent, external validity is non-establishable.
•
If we cannot determine some confidence in the validity of the independent variable
affecting participants in some way, it is basically impossible to generalize this “effect” to
other people / scenarios.
Ecological validity of RCT´s:
Ecological validity is a subset of external validity focusing on whether the results of a study
can be generalized to real-life settings. This includes the relevance and applicability of the
research context to everyday life. RCTs often have lower ecological validity due to:
1. Artificial Environment: The controlled settings in which RCTs are conducted may
not mimic the complexities of real-world environments.
2. Participant Behavior: Knowing that they are part of a study, participants might alter
their behavior (Hawthorne effect), which would not occur in everyday life.
3. Intervention Implementation: The way interventions are delivered in RCTs may be
more structured and intensive compared to usual practice.
Experiments:
Experiments is a category of research designs which includes several sub-categories.
In experiments in-general, we wish to manipulate one or more independent variables to
observe the effect they have on a dependent variable while we are very observant about
possible influencing factors and biases.
We for example, want to view the effect of 2 to 3 (or more) levels of change, (say exposure
to something) and how and increase/decrease in exposure affect the dependent variable.
Experiments are not per-se common in social sciences due to the need for manipulations of
our variables. It can be hard to change the gender / beliefs / motivation of people, simply to
view the effect this has on some dependent variable. Some cases do however enable
experiments in social sciences which can yield powerful results.
Experiments: Threats to internal and external validity:
Threats to internal validity in case of no RCT:
Campbell (1957) and Cook and Campbell (1979).
Testing: The possibility that subjects may become sensitized to the aims of the experiment
(see Research in focus 3.7). RCT allows us to discount this possibility if there is no difference
between the experimental and control groups.
History: This threat refers to the possibility that events in the experimental environment
that are unrelated to manipulation of the independent variable may have caused the
changes.
Maturation: Quite simply, people change, and the ways in which they change may have
implications for the dependent variable.
Selection: If there are differences between the two groups, which would arise if they had
been selected by a non-random process, variations between the experimental and the
control groups could be attributed to pre-existing differences in their membership.
Ambiguity about the direction of causal influence: The very notion of an independent
variable and a dependent variable presupposes a direction of causality. However, there may
be occasions when the temporal sequence is unclear, so that it is not possible to establish
which variable affects the other.
Threats to external validity in case of no RCT:
Campbell (1957) and Cook and Campbell (1979).
Interaction of selection and treatment. This threat raises the question: to what social
and psychological groups can a finding be generalized?
Interaction of setting and treatment: This threat relates to the issue of how confident we
can be that the results of a study can be applied to other settings.
Interaction of history and treatment: This raises the question of whether or not the
findings can be generalized to the past and to the future.
Interaction effects of pre-testing: As a result of being pre-tested, subjects in an
experiment may become sensitized to the experimental treatment. Consequently, the
findings may not be generalizable to groups that have not been pre-tested and, of course,
in the real world, people are rarely tested in this way.
Reactive effects of experimental arrangements: People are frequently, if not invariably,
aware that they are participating in an experiment. Their awareness may influence how
they respond to the experimental treatment and therefore affect the generalizability of the
findings.
Ecological validity of experiments: High for field experiments due to real-life settings. Low
for lab-experiments due to unnatural settings.
Classic experiment:
Consist of two groups of participants. One group which receives the treatment, another
group which does not (control group). This increases the internal validity of the research
results.
If the study starts off with a single sample group, which subsequently is randomly
distributed into a treatment group and a control group, this is called an RCT (Randomized
controlled trial).
Lab experiments:
Lab experiments are conducted in a very controlled environment, often a laboratory setting
where researchers have a high degree of control of the situation and setting and are
thereby in more power of the variables than alternative experimental designs.
Lab experiments are characterizes as being in high control which allow researchers to
isolate for the effect of independent variables better. To better control the sample
population, researchers often assign the population randomly into groups (RCT), to
minimize possible biases.
Advantages of lab experiments are a high degree of internal validity because researcher
are in high control of variables and because it is easier to replicate.
Disadvantages of lab experiments is mainly that it is difficult to generalize to real world
setting. Meaning low external validity. Also ecological validity can be low because the study
may not be sufficiently naturalistic.
Lastly, awareness bias often occurs in lab experiments. This is where participants alter their
behaviour or answers because they are aware that they are being studied.
Field experiments:
Field experiments are, opposite of lab experiments conducted in real-world settings where
the environment is not controlled or is trying to replicate a real-world setting perfectly.
In field experiments researchers manipulate independent variables and try to measure and
analyse their effects on the dependent variable within the natural context.
Advantages of field experiments: They provide a much higher ecological validity, as they
are conducted in real-world settings. Also, field experiments better allow researchers to
observe natural behaviour and contexts.
Disadvantages of field experiments: They are challenging to replicate and lower internal
validity as researchers cannot control variables to the same extend.
Quasi experiments:
Quasi experiments share many characteristics with other experiments which follow the
rules of experiments more stringent. Quasi experiments, however, do not randomly assign
participants into groups.
Researchers take advantage of naturally occurring groups or settings, in order to study the
effects of the independent variables. We don’t randomly assign participants into groups
mainly because of ethical or social considerations.
The advantage of quasi experiments is to investigate cause-and-effect relationships in
situations where “true” experimental designs is not possible or ethically correct.
The disadvantage of quasi experiments is that researchers have less control over
potential confounding variables compared to true experiments, and this can be a threat to
the internal validity of the research design.
Internal validity: Low due to quasi experiments being less effective at controlling
extraneous variables, relative to lab experiments.
External validity: Higher than lab-experiments, lower than field experiments. Depends
greatly on the representativeness of the sample and the reliability of the measures.
Ecological validity: High due to quasi experiments often being conducted in naturalistic
settings. Can be low if they are not.
Case study:
In case studies the concept is to introduce an in-depth examination of a single individual,
group, ideology, or event. It aims at providing detailed context-rich descriptions and
analyses of the subject. In case studies, researcher gather data through various methods,
often qualitative, like interviews or document analysis. Researchers emphasize
understanding complexities and nuances of the specific case.
Case studies are unique in their results because they generally produce very specific and indepth analyses on specific events which is advantageous. This can be extremely valuable
for hypothesis-generation, but also present the disadvantage of being difficult to
generalize and replicate in the future.
Internal Validity: Medium to High: In case studies, internal validity can be relatively high if
the case is well-chosen and well-documented. Detailed, in-depth data collection and using
multiple sources of evidence can strengthen internal validity. However, the absence of
control groups and the potential for researcher bias can lower internal validity.
External Validity: Low to Medium: Case studies typically focus on a single case or a small
number of cases, making it difficult to generalize findings broadly. However, if the case is
representative of a larger population or phenomenon, the external validity can be medium.
Ecological Validity: High: Case studies often involve observing and analyzing phenomena in
their natural contexts, which can provide a high degree of ecological validity. The detailed,
context-rich nature of case studies allows for a deep understanding of the phenomenon
within its real-world setting, making the findings highly relevant to similar real-life situations.
Cross sectional design:
Cross-sectional research designs, also called snap shots or correlation studies, focus on
collecting data from a study-population at a specific point in time. The aim of crosssectional design is to understand relationships between the variables which are being
studied but without manipulating them.
In cross-sectional research design more than one case is being studies.
Cross sectional design is often carried out through surveys, observations, or tests, where we
capture a snapshot of the characteristics of the variables.
Cross sectional design is constructed much like longitudinal design, but we study the
included variables only once. We thereby can infer correlation but lack inference about
causation in cross sectional design.
Advantages:
Efficiency: Cross-sectional studies are time-efficient since they collect data at a single point
in time, avoiding the long durations associated with longitudinal studies.
Cost-Effectiveness: These studies are generally less expensive to conduct because they do
not require follow-up over time.
Variation and Comparisons: By including more than one case, cross-sectional studies can
capture variation and enable comparisons between different groups, organizations, or
entities .
Wide Range of Methods: They can incorporate various research methods such as surveys,
structured observations, content analysis, and the use of official statistics, providing
flexibility in data collection.
Generalizability: When using a representative sample, the findings can often be
generalized to the larger population, enhancing external validity .
Disadvantages:
Causality Issues: A significant limitation is the inability to establish causality due to the lack
of temporal order between variables. This makes it difficult to determine the direction of
causal relationships.
Internal Validity: Cross-sectional designs often suffer from low internal validity because
they cannot definitively establish cause-and-effect relationships .
Ecological Validity: The use of structured instruments like questionnaires may disrupt the
natural setting, potentially affecting the ecological validity of the findings .
Snapshot Limitation: They only provide a snapshot of the variables at one point in time,
which may not capture changes or developments that occur over time .
Internal, External, and Ecological Validity
Internal Validity: Cross-sectional designs typically have weak internal validity because
they cannot establish a clear causal direction due to the simultaneous collection of data on
multiple variables. This limitation arises because the researcher cannot manipulate the
variables or control the timing of data collection, leading to ambiguity in causal inferences .
External Validity: External validity is generally strong in cross-sectional studies,
particularly when data are collected from a randomly selected, representative sample. This
allows for generalization of the findings to the broader population. However, if non-random
sampling methods are used, external validity can be compromised .
Ecological Validity: The ecological validity of cross-sectional studies may be jeopardized
by the use of structured research instruments, such as questionnaires and structured
observation schedules. These tools can disrupt the natural environment of the subjects,
leading to concerns about the naturalness and realism of the data collected .
Relevant References and Quotes
Internal Validity: "Cross-sectional research designs produce associations rather than
findings from which causal inferences can be unambiguously made. Procedures for making
causal inferences from cross-sectional data will be referred to in Chapter 15, though most
researchers feel that the resulting causal findings rarely have the internal validity of those
deriving from experimental designs" .
External Validity: "External validity is strong when, as in the case of the SAL, the sample
from which data are collected has been randomly selected. When non-random methods of
sampling are employed, external validity becomes questionable" .
Ecological Validity: "Since much cross-sectional research uses research instruments, such
as self-completion questionnaires and structured observation schedules, ecological validity
may be jeopardized because these very instruments disrupt the 'natural habitat'"
Longitudinal:
Longitudinal design involves studying specific individuals, groups, or events over an
extended period of time. We often want to characterize changes or developments in the
variables which we study. Longitudinal designs are conducted much like cross sectional
design where data is collected via observations, surveys, or tests where researchers look for
trends in the variables.
Longitudinal designs allow researchers to both infer about correlation and causation
through analysing changes and trends.
Comparative:
Comparative design involves comparing two or more groups or settings to characterize
similarities and differences, patterns, and developments. Researchers want to select data
on several variables of interest of at least two groups or situations. Comparative design
differs from case study design in that we do not look at a single group or setting, but two or
more, and from that we are able to make comparative analysis and developmental
depending on if data is collected at one point in time or over a period of time.
Scientists and their choice of paradigm:
Popper:
Popper has strong opposition towards positivism. He considers it as a failed paradigm. He
instead leans more towards his own type of paradigm which he characterizes as critical
rationalism and realism. (ABH chapter 3).
Acc. To Popper, scientific theories can only be falsified, not proven, which means they
should be subject to constant testing and improvement. He thinks scientists are important
because they continually edge closer towards the truth, not because they arrive at any
definitive truth. (Popper 1975, 356).
This view differs from positivism, where we believe scientific knowledge is based only on
observable facts and a measurable reality.
Post-positivism therefore fits Poppers scientific beliefs the best as post-positivism
maintains a high value of empirical testing and objectivity, while also acknowledging the
limitations of strict empiricism and advantages of interpretation and subjectivity.
Kuhn:
Looking at Kuhns theories, there are no pre-existing models to his notion of scientific
paradigm, suggesting a constructionist view of the world, according to the definition from
Guba (1990).
Kuhn actively shares criticism of the positivistic view, noting “There is, I think, no theoryindependent way to reconstruct phrases like ‘really there’; the notion of a match between
the ontology of a theory and its ”real” counterpart in nature now seems to me illusive in
principle” (Kuhn 1962, p. 206) Kuhn does not believe in the notion of being fully objective in
the research. Additionally, his description of how people react to stimuli suggest that he
does favor a more constructivist view.
”If two people stand at the same place and gaze in the same direction, we must…conclude
that they receive closely similar stimuli…But people do not see stimuli…Instead they have
sensations, and we are under no compulsion to suppose that the sensations of two viewers
are the same…much neural processing takes place between the receipt of a stimulus and
the awareness of a sensation…the route from stimulus to sensation is in part condition by
education.” (Kuhn 1962, p. 192-193) The above quote suggests that Kuhn sees meaning as
something people construct for themselves (constructivist) rather than an objective
reality(positivist).
In sum, it is believed that Kuhn favor a constructionist view on the world, where theory is
formed based on subjective understanding of the world informed by previous theory.
Kahneman:
Best characterized as a post-positivist.
Empirical and Scientific Approach: Like positivism, post-positivism emphasizes the
importance of empirical evidence and scientific methods. Kahneman's work is grounded in
experimental psychology, using rigorous empirical methods to study human behavior. This
scientific approach aligns well with post-positivist principles.
Recognition of Human Limitations: Post-positivism acknowledges that human
understanding is inherently limited and that our observations are fallible. Kahneman's
research on cognitive biases and heuristics directly addresses the limitations and
imperfections of human cognition, challenging the notion of perfect rationality that
traditional positivism often assumes.
Critical Realism: Post-positivism adopts a stance of critical realism, which accepts that
while an objective reality exists, our understanding of it is mediated by human cognition
and social factors. Kahneman's work recognizes that while there are patterns and
regularities in human behavior, our perceptions and decisions are influenced by cognitive
biases and contextual factors.
Theory Revision and Falsifiability: Post-positivism emphasizes the ongoing revision of
theories in light of new evidence. Kahneman's work has led to the development of new
theories (e.g., Prospect Theory) that challenge and refine existing economic models. His
approach is consistent with the post-positivist view that scientific knowledge is provisional
and subject to refinement.
Integration of Multiple Perspectives: While not strictly constructivist, post-positivism is
more open to integrating insights from various disciplines compared to strict positivism.
Kahneman’s integration of psychological insights into economic models is a hallmark of this
interdisciplinary openness.
Nelson:
In general, I would argue the way he addresses quantifiable measures makes him seem like
a post-positivist, as it seems as though some data/numbers do have a face value on their
own, such as in the following quote: “The numbers used in the social and behavioural
sciences almost always are, by themselves, somewhat limited and imprecise characterizers
of the phenomena they are designed to measure, and need to be understood as part of a
broader and more detailed if qualitative characterization.” (Nelson 2016, page 1695).
However, the way he addresses that data needs to be understood as part of a broader
qualitative characterization makes him seem more constructivist, as it seems he relies more
on the creation between the data and the researcher to create meaning.
To conclude, I would argue that Nelson’s 2016 paper is definitely not positivistic, as he does
not believe theories are a window of the one “true” reality. However, when evaluating
whether he is post-positivistic or constructivist, I would argue that the discussion gets
blurrier, and that he is somewhere in between.
Guba:
Guba states himself that he is a constructivist.
Constructivism: Reject objectivity, celebrate subjectivity.
He is also relativist:
refers to the idea that knowledge and truth are not absolute but are instead shaped by
social, cultural, historical, and contextual factors. This perspective challenges the notion
that science is purely objective, or that scientific knowledge can be understood outside of
the context in which it is developed.
Richie:
Ritchie's epistemology aligns with a combination of post-positivism and constructivism.
Post-positivism shares many beliefs with positivism, the idea that our research can be
objective, value-free and independent of context. But unlike positivists, post-positivists
believe that our knowledge is always provisional and subject to revision when new evidence
emerges. Constructivism believes that our understanding of reality is always partial and
subjective, and we construct it through our experiences and interactions with the world.
Therefore, Ritchie combines post-positivism's critical approach with the constructivist's
belief in a subjective reality. (Ritchie 2020 ch 1, p. 21-22)
Evaluate research quality: Quantitative and Qualitative
Quantitative research evaluation:
The three criteria for evaluating quantitative research:
1) Reliability
2) Replicability
3) Validity
a. Internal validity
b. External validity
c. Ecological validity
Qualitative research evaluation:
List of Biases, thorough explanation:
Selection bias:
Occurs when the selection of participants into a study is not random, leading to a nonrepresentative sample that can skew results.
Example:
If we wish to study the likely hood of the chance of member of a population to get cancer, it
would be very wrong of us, as researchers, to go the cancer-wing in the hospital to collect
data. All datapoints would indicate cancer, and we would as such say that the chance is
100%, simply because we did not pick a representative population for the study.
Omitted variable bias:
Occurs when an important variable is left out of the model. This will lead to incorrect
estimates of the effect of other variables included in the model.
Example:
If we wish to study the reason why one person’s bank account has more money in it than
another person’s, we could include variables like: Education, age, gender, social class. If we
however leave out the specific job or income of the two people, this would be omitting a
very important variable.
Publication bias:
Refers to the tendency of researchers to publish positive results more frequently than ‘null’,
or negative results. This can distort the overall picture of a research area.
Example:
A researcher studying the education system has made several studies over the last decade,
each study that were aligned with his hypothesis was published, while he did not publish
the null or negative hypothesis research’s… This has caused the information on the subject
to be positively skewed.
Confirmation bias:
Favouring information which confirms ones pre-existing beliefs or hypotheses, while
disregarding contradictory information.
Explanation:
Confirmation bias is closely linked to publication bias, but while publication bias is more
regarding what actually gets published, confirmation bias regards researchers’ choices of
studies.
Sampling bias:
When the selection of participants is biased, resulting in a non-representative sample
group.
Example:
Researcher’s wish to study the Danes understanding of monetary policy and the
implications for the economy of different economic interventions by the national bank. The
researcher’s sample from BSS and CBS only, causing a skewed knowledge result.
Response bias:
When participants answers are affected by social desirability, leading questions, or nonanonymous publication of answers. This can cause inaccurate data.
Example:
A questionnaire to be answered by a group of people is made to answer during a lecture.
Each person’s answer is shown on the projector screen with answer and name of
participant.
Recall bias:
When participants ability to recall past events is influenced by time-past or the emotional
state of participants, either in the current situation or in the situation they are trying to
recall.
Example:
It can be hard to recall what happened in a situation, where you were very upset or afraid
because the brain sometimes represses those memories because it was a traumatic
experience.
Survivorship bias:
When only survivors or successes are included in the narrative, which can lead to inaccurate
actions or overly positive research conclusions.
Example:
Fighter jets returning from war with bullet holes. Where to reinforce armour?
Experimenter bias:
Occurs when the researcher’s beliefs or expectations influence the outcome of the study
either consciously or unconsciously.
Example:
Researcher has a pre-existing belief about the actuality of a study and therefore pick and
choose research papers that support this belief, while disregarding evidence on alternative
information.
Attrition bias:
Occurs when participants drop out of the study, leading to a non-representative sample and
potentially affecting the study’s results.
Time-order bias:
This occurs when the order of events is incorrectly interpreted, leading to incorrect
conclusions about causation.
Cross sectional bias:
Occurs when data collected at a single point in time is incorrectly interpreted as showing
causal relationships.
Example:
Researchers collect data on smoking habits in 2020 and make the mistake of saying that
this shows an increasing trend in smokers. This cannot be true as there has only been
collected one sample.
Proxy measurement bias:
Occurs when a variable is used as a proxy for another variable, but the proxy does not
accurately represent the original variable.
Example:
Researchers want to figure out why college students love Nike-shoes. Buying a bunch of
authentic Nike shoes for people to test is however too expensive for the researchers, so
they instead buy very poorly constructed fake Nikes.
Cultural bias:
Occurs when cultural differences are not considered in the design of the study or the
interpretation of the results.
Examples:
Researchers want to study why Scandinavian countries consume more beef than people in
India. Forgetting that cows are sacred animals in India.
Volunteer bias:
Occurs when participants who volunteer for a study are different from those who do not,
leading to a non-representative sample.
Example:
Study on happiness, where participants are only chosen if they volunteer. Volunteers might
be happier (in the moment) which is why they volunteer, causing the results to possibly be
overly positively skewed.
Reporting bias:
Occurs when researchers selectively report only certain results from a study, leading to a
distorted picture of the overall findings.
Example:
Positive effects from cannabis smoking is omitted in the report, because it did not fit the
intention, belief, or design of the researchers study.
Instrument bias:
Occurs when the instrument used to measure a variable is not valid or reliable, leading to
inaccurate data.
Example:
Researchers set up a study where they want to find the perfect time for baking a specific
cake to some pre-determined standards. They use a clock the varies in quality of timekeeping.
Measurement bias:
Occurs when the way a variable is measured changes over time or across different groups,
leading to inconsistent results.
Example:
Instruments for measuring some variable may become much more accurate due to a
technological innovation, causing older data to be more questionable.
Confounding: (Third variable problem)
When looking at the relationship between two variables and wrongfully thinking that one is
affecting the other / they both affect each other, when in fact a third “hidden” variable is
causing the change in both factors.
Example:
Eating ice cream cause people to be happier.
More ice-cream = better mood.
People at a park I surveyed, and the results seem to support the hypothesis.
Weather / temperature is however a variable that has been left out. When the temperature
is considered in the research, we find that better weather cause both better mood and
more ice cream consumption.
Weather / Temperature is as such a confounding variable.
Causality and correlation:
Look at Isager 2023 text notes for explanations.
The scientific method:
The scientific method is used in causal inference to falsify all the plausible but wrong
models until only the one true causal model remains. It involves imagining all causal models
that could plausibly explain the data, and then using empirical observations to test and
refine those models. Reference: Page 8-9. (Isager 2023)
Different views on key subjects:
Paradigm beliefs: Richie 2020 - Watts 2011, Popper 1962 - Kuhn 1962
Points of agreement:
Richie 2020 and Watts 2011: Both sources agree on the importance of paradigms in
guiding scientific research, though they may define and approach paradigms differently.
Popper 1962 and Kuhn 1962: There is a consensus that scientific paradigms shape the
direction and interpretation of research, although Popper focuses on falsifiability while
Kuhn emphasizes the role of paradigms in normal science and revolutionary shifts.
Points of disagreement:
Popper vs. Kuhn: Popper's emphasis on falsifiability as the core of scientific progress
contrasts with Kuhn's view of science advancing through paradigm shifts and revolutions.
Guba 1990 vs. Positivism/Post-Positivism: Guba criticizes positivist approaches for being
too rigid and promotes alternative paradigms that accommodate more flexible and
interpretative methodologies.
Karl Popper:
Paradigmatic Beliefs:
• Falsifiability: Popper argues that scientific theories cannot be proven but can only
be falsified. A theory should make bold predictions that can be tested and
potentially refuted by observations and experiments.
• Critical Rationalism: Popper's approach emphasizes critical scrutiny and the
continuous testing of hypotheses. He believes that scientific progress is made
through the process of conjectures and refutations.
•
Rejection of Induction: Popper challenges the traditional inductive methods in
science, suggesting that science advances through deductive testing of hypotheses
rather than accumulating observational data.
Yvonna S. Lincoln & Egon G. Guba (often discussed together due to their
collaborative work):
Paradigmatic Beliefs:
• Alternative Paradigms: Guba and Lincoln argue against the dominance of
positivism and propose alternative paradigms such as constructivism, which
emphasizes the subjective construction of reality.
• Ontological and Epistemological Pluralism: They advocate for a pluralistic
approach to ontology (nature of reality) and epistemology (nature of knowledge),
recognizing multiple ways of knowing and understanding the world.
• Constructivism: This paradigm asserts that knowledge is co-constructed between
researchers and participants, and reality is seen as a social construct rather than an
objective entity.
Richard R. Nelson:
Paradigmatic Beliefs:
• Evolutionary Economics: Nelson's work in economics is often associated with an
evolutionary approach, which sees economic processes as dynamic and evolving
rather than static and equilibrium-based.
• Differences in Scientific Fields: Nelson emphasizes that different scientific fields
operate under different paradigms and methodologies, highlighting the unique
characteristics and challenges of each field.
• Institutional Influence: He acknowledges the significant role of institutions in
shaping scientific research and technological innovation, arguing that institutional
contexts influence the direction and nature of scientific progress.
Comparison and Contrast:
• Falsifiability vs. Constructivism: Popper's emphasis on falsifiability as a criterion for
scientific theories contrasts sharply with Guba and Lincoln's constructivist
paradigm, which rejects the idea of an objective reality that can be falsified.
• Objective Reality vs. Subjective Reality: Popper maintains that science aims to
uncover objective truths about the world, whereas Guba and Lincoln argue for the
existence of multiple subjective realities constructed through social interactions.
• Methodological Rigor vs. Methodological Pluralism: Popper advocates for
rigorous, deductive testing of hypotheses, while Guba and Lincoln promote
methodological pluralism, accepting various qualitative and interpretative
approaches as valid.
• Role of Institutions: Nelson's focus on the role of institutions in shaping scientific
research differs from both Popper's and Guba's perspectives, adding a dimension of
socio-economic context to the discussion of paradigms.
Points of Agreement:
•
•
Importance of Paradigms: All three recognize the crucial role paradigms play in
guiding scientific research and understanding. They agree that paradigms shape
how scientists view the world and approach their work.
Scientific Progress: Despite their differences, all three scholars discuss the concept
of scientific progress, though they explain it through different lenses (falsifiability,
constructivism, evolutionary change).
Points of Disagreement:
• Nature of Scientific Theories: Popper views scientific theories as conjectures to be
tested and falsified, while Guba and Lincoln see theories as constructs that emerge
from social interactions and shared understandings.
• Objective vs. Subjective Reality: Popper's objective realism contrasts with the
subjective, co-constructed realities proposed by Guba and Lincoln.
• Methodological Approaches: Popper's preference for rigorous, quantitative
methods is at odds with Guba and Lincoln's acceptance of qualitative, interpretive
methodologies.
Theory and its nature:
Agreement between Bergenholtz and Kahneman.
Bergenholtz 2024c: highlight the essence of theory in providing explanatory frameworks for
understanding phenomena.
Kahneman's Thinking Fast and Slow: Agrees with the notion that theories need to be
grounded in empirical research to provide valid insights.
Scientific change:
POS chap. 5 and Popper 1962: There is an agreement on the concept of scientific
revolutions and how new paradigms replace old ones in the course of scientific progress.
Guba 1990 and Nelson 2016: Both emphasize the evolving nature of scientific paradigms
and how shifts in paradigms reflect deeper changes in scientific understanding.
Scientific change and theory:
Disagreements:
Kuhn 1962 vs. Popper 1962: Kuhn’s idea of incommensurability, which suggests that
different paradigms cannot be compared directly, contrasts with Popper's view that
scientific theories should be continuously tested and falsified.
Sullivan 2011 and Watts 2007: Different perspectives on embracing complexity in scientific
research, with Sullivan advocating for more comprehensive approaches while Watts
focuses on streamlined methodologies suitable for the 21st century.
Points of Agreement:
1. Importance of Paradigms:
o Guba & Lincoln: "Paradigms represent a basic set of beliefs that guide
action." (Guba & Lincoln, 1994)
o Popper: "The history of science, like the history of all human ideas, is a
history of irresponsible dreams, of obstinacy, and of error." (Popper, 1963)
2. Scientific Progress:
o Despite their differences, all three scholars discuss the concept of scientific
progress, though they explain it through different lenses (falsifiability,
constructivism, evolutionary change).
Points of Disagreement:
1. Nature of Scientific Theories:
o Popper: "A theory which is not refutable by any conceivable event is nonscientific. Irrefutability is not a virtue of a theory (as people often think) but a
vice." (Popper, "The Logic of Scientific Discovery," 1959)
o Guba & Lincoln: "There is no single 'reality' or 'truth,' but rather multiple
constructions of reality that are shared and co-constructed through
interaction." (Guba & Lincoln, 1985)
2. Objective vs. Subjective Reality:
o Popper: "Science seeks to approximate truth by eliminating errors." (Popper,
1963)
o Guba & Lincoln: "Reality is subjective and constructed through human
interaction and interpretation." (Guba & Lincoln, 1985)
3. Methodological Approaches:
o Popper: "Science must proceed by falsification rather than verification."
(Popper, 1959)
o Guba & Lincoln: "Qualitative methods are necessary to understand the
depth and complexity of human experiences and social phenomena." (Guba
& Lincoln, 1994)
Detailed Analysis:
Karl Popper:
Paradigmatic Beliefs:
•
•
Falsifiability: Popper argues that scientific theories cannot be proven but can only be
falsified. A theory should make bold predictions that can be tested and potentially
refuted by observations and experiments.
o Quote: "A theory which is not refutable by any conceivable event is nonscientific. Irrefutability is not a virtue of a theory (as people often think) but a
vice." (Popper, "The Logic of Scientific Discovery," 1959)
o Reference: Popper, K. R. (1959). The Logic of Scientific Discovery. Hutchinson
& Co.
Critical Rationalism: Popper's approach emphasizes critical scrutiny and the
continuous testing of hypotheses. He believes that scientific progress is made through
the process of conjectures and refutations.
o Quote: "Science must begin with myths, and with the criticism of myths."
(Popper, "Conjectures and Refutations," 1963)
o Reference: Popper, K. R. (1963). Conjectures and Refutations: The Growth of
Scientific Knowledge. Routledge.
Yvonna S. Lincoln & Egon G. Guba:
Paradigmatic Beliefs:
•
•
Alternative Paradigms: Guba and Lincoln argue against the dominance of
positivism and propose alternative paradigms such as constructivism, which
emphasizes the subjective construction of reality.
o Quote: "Constructivism, while certainly not a monolithic or entirely unified
set of philosophical beliefs, is characterized by a belief in the social
construction of reality." (Guba & Lincoln, "Competing Paradigms in
Qualitative Research," 1994)
o Reference: Guba, E. G., & Lincoln, Y. S. (1994). Competing Paradigms in
Qualitative Research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of
Qualitative Research (pp. 105-117). Sage Publications.
Ontological and Epistemological Pluralism: They advocate for a pluralistic
approach to ontology (nature of reality) and epistemology (nature of knowledge),
recognizing multiple ways of knowing and understanding the world.
o Quote: "The basic beliefs of these paradigms can be summarized by the
responses given to three fundamental questions: the ontological, the
epistemological, and the methodological question." (Guba & Lincoln,
"Naturalistic Inquiry," 1985)
o Reference: Guba, E. G., & Lincoln, Y. S. (1985). Naturalistic Inquiry. Sage
Publications.
Richard R. Nelson:
Paradigmatic Beliefs:
•
•
Evolutionary Economics: Nelson's work in economics is often associated with an
evolutionary approach, which sees economic processes as dynamic and evolving
rather than static and equilibrium-based.
o Quote: "Economic change is largely a process of continual evolution, where
technologies and institutions co-evolve in a complex interplay." (Nelson, "An
Evolutionary Theory of Economic Change," 1982)
o Reference: Nelson, R. R., & Winter, S. G. (1982). An Evolutionary Theory of
Economic Change. Belknap Press.
Differences in Scientific Fields: Nelson emphasizes that different scientific fields
operate under different paradigms and methodologies, highlighting the unique
characteristics and challenges of each field.
o Quote: "Different scientific fields develop different styles of research and
approaches to knowledge, influenced by their specific historical, social, and
institutional contexts." (Nelson, "The Sources of Economic Growth," 1996)
o Reference: Nelson, R. R. (1996). The Sources of Economic Growth. Harvard
University Press.
Randomized control trials: Nelson and Watts opinions:
Nelson 2016 has 3 arguments regarding why social science differ from traditional science.
-
Different subjects being examined, where many physical phenomena like a
certain cell will always look the same. This is rarely true for social sciences.
RCT results will thus not make sense to generalize because the subjects’
studies will not be homogenous with what you try to generalize towards.
-
Influential variables play a large role in the heterogeneity of social studies.
“Second, the forces and conditions that influence the subjects of study in
ways that the science seeks to understand are numerous, highly variable,
and often cannot be separated sharply one from another. Indeed, the
circumstances associated with any particular set of observations may need
to be understood as, in some sense, unique” (Nelson 2016, page 1697).
As we in RCT´s wish to isolate the effects of one variable, this quote goes
directly against the applicability of RCT in social sciences due to innate
interconnectivity between variables.
-
Subjects being studied change over time. This does not go against RCT´s perse, but it does state that we need longitudinal data in order to account for
this changing environment. Even when you try to divide participants into
finer sub-categories, Nelson states that often, large heterogeneous
tendencies still show. This goes against RCT. “Science struggles with this
heterogeneity by trying to divide up the variety into sub-classes that are
more homogenous. But the history of such research shows that almost
always there continues to be considerable heterogeneity within even the
finer disease classifications.” (Nelson 2016, page 1699).
Watts 2011, RCT´s
Watt's 2011 paper aligns closely with Nelson's arguments about the differences between
social sciences and traditional sciences. Watt explores the complexities within social
sciences, contending that they are less straightforward than often perceived. He provides
two main reasons for this. First, he highlights the difficulty in tracing variables across
different scales, noting that "social systems... exhibit 'emergent' behaviour," making it
challenging to relate behaviors at one level to properties at a different level (Watts, 2011, p.
31). For instance, firms maintain stable identities despite changes in employees, much like a
person's identity remains consistent even as their body cells are replaced.
Watts argues that this emergent behavior complicates the reliability of Randomized
Controlled Trials (RCTs) in social sciences, as the same study might yield different results
under varying conditions. His second reason is that social scientists must consider multiple
scales simultaneously, unlike in physics where isolated variables can be studied: “Unlike in
physics, therefore, essentially every problem of interest to social scientists requires them to
consider events, agents and forces across multiple scales simultaneously” (Watts, 2011, p.
32). This perspective mirrors Nelson's view that social science problems cannot be
adequately addressed through RCTs due to the multitude of variables involved.
In his 2007 paper, Watts elaborates on the inherent difficulties of social science, which stem
from the numerous relevant variables and their interconnectedness. He introduces the
concept of network analysis to quantify these interactions: “It is hard to understand, for
example, why even a single organization behaves the way it does without considering (a)
the individuals who work in it; (b) the other organizations with which it competes,
cooperates and compares itself to; (c) the institutional and regulatory structure within
which it operates; and (d) the interactions between all these components.” (Watts, 2007, p.
489). He emphasizes that understanding a single organization requires evaluating all these
factors simultaneously due to their interrelated nature.
Ultimately, both Nelson's 2016 and Watts' 2007 and 2011 papers challenge the notion that
RCTs are the gold standard for scientific research in social sciences. They argue that the
field's heterogeneity makes it nearly impossible to isolate single variables effectively, as
comprehensive understanding necessitates considering multiple interconnected factors.