Discussion about exploratory vs. confirmatory research.
Definition:

EJ
o
o

Exploratory research uses the data twice basically.
Confirmatory research uses the data to test hypotheses, almost only happens in
medicine…
Denny
o Exploratory research is a non-complete set of procedures, building theory.
o Confirmatory research can make use of pre-registration. Pre-registration is
however the method, not the definition of confirmatory research. Almost no
research in science is purely confirmatory right now.
Educating current and future researchers about exploratory versus confirmatory research will
greatly reduce the number of people presenting exploratory research as confirmatory.
EJ: Definitely a good idea to educate on the difference! This should have its place early on in
scientific education; it should be taught in the first year actually! It is at least good that there is a
course like Good Science Bad Science now. Ideally, all research is clearly defined, and research
with low power is not sensible. But it would be unfair of me to demand a high sample size in all
research for example, because this is often not feasible.
D: The difference is quite trivial. It’s hard to say how teaching this will turn out. There is always a
continuum between the two types. ‘Science’ is a very big ‘field’ and psychology has a lot of
different paradigms. There should be a “hardcore” of science. Genetics for instance, solved its
own problem by only moving to publication when replication is confirmed. It’s important to
handle data carefully.
Who is in your opinion most responsible for the fact that people do exploratory analyses and
report them as confirmatory?"
D: it’s about the whole context in which this happens, it is a sort of network, everybody does it
and everybody is responsible to some extent. Besides, it’s not true that exploratory research (or
non-significant research!) won’t get published, people just don’t try it, they are stuck in their
beliefs, stuck in the habits of the system. It’s basically bad science, about incredible theories,
creating a wow effect. But if its truly good stuff and you find nothing, it will be published!
EJ: Intermediate results are a problem: when there is no real clear effect but also no absence. If
the evidence it’s not convincing/ambivalent you better try again. 0.06 is marginally significant;
it shows a ‘trend’. Reviewers then tell us to do a second experiment.
D: Peer reviewing sucks. Most of the time they are just inadequate at judging the quality of a
paper and bring their own subjectiveness to the table.
EJ: People are actually rooting for getting rid of journals. On the other hand, my papers get better
through feedback. A solution to peer review could be to have no review at first. First put the
paper on an open archive, sort of like pre-registering it and only after that let it be officially
reviewed.
D: Reviewing clouds science. Reviewers make the paper better even though the writer doesn’t
understand what he is doing. So you are actually reading a paper that is a combination of the
work and ideas of the original writer ànd the reviewer. There is a lot of nonsense out there.
Do different fields in psychology differ in the exploratory vs confirmatory research debate?
EJ: In social psychology they overestimate the extent to which they have theory --> we should
have more theory --> but people who claim this are not able to make good theory. A good theory
is a theory that extents to a model that makes predictions and can be directly tested. We should
therefore move to theory that is more concrete either by using simulations or mathematical
models. In certain areas, e.g. FMRI, it’s very expensive to collect data so it’s much more difficult
to replicate (your own) research. Some fields also have smaller effect sizes, they might be more
likely to “tweak” their data. Furthermore, within subject studies have higher power! it’s the
between-subject experiments that are noisy --> within subject studies are more likely to be
replicated.
D: We don’t have a culture where theory construction is seen as a skill, a lot of people lack the
ability of “theory construction”. In other scientific fields, e.g. physics, biology etc, theoretical
science and practical science is clearly separated, people who construct theory don’t go into the
field --> but this is not the case in psychology.
How do we move psychology in the direction of theory construction?
Psychology should have a broader focus. How do you know how to construct good theories if
you don’t even know what good theories are? We should ask ourselves what the goal of science
is? What should we try to achieve?
Should a good researcher be measured by the amount of publications they have?
EJ: The publication criterium is good for people who would otherwise be doing nothing. It’s
difficult because you do need to have some sort of incentive. In the current publication system,
at least the criteria are clear, that’s the advantage of publication criteria, so there’s no “bullying”
from the alpha monkey. We need systems that can reward both individuals and groups. The
current system does reward performance and merit…
Would you recommend being a scientist?
EJ/D: The biggest advantage of not being a scientist is that when you go home at 5pm you are
done with you work – but not if you are a scientist. People work at night and in the weekend.
Somebody said: in an ideal world, we should be able to be a good scientist with 40 hours work a
week. but that’s not the case. Science is on your mind all the time --> it’s like running your own
company. It looks like you have a lot of freedom, but at the same time you don’t. YOU are
completely responsible for your own success. Maybe its comparable to a musician or artist -->
it’s not a job, it’s a lifestyle.
What is the most important skill for a psychologist/scientist?
EJ/D: Thinking is the most important skill! Statistics is not the most important, it is just an
indicator of a certain way of thinking, it is a symptom, not a cause of a good science. Another
important skill is creative simulation! Thinking about what your theory does when it is
concretely implemented. Another skill: writing! Enjoying writing is quite important. If you find it
really difficult to express your ideas, it will be very difficult for you to survive in the current
academic climate. You should have fun with what you’re doing!
D: Basis of science: we are human, limited and need to acknowledge our mistakes and learn from
them! You will fall, and then you have to get up. A crisis in psychology is a good thing! Being
critical means we are in a state of improvement. Like Francis Bacon said: We are human and
limited, acknowledge this and accept it and learn from our mistakes!
EJ: We are witnessing a transitional stage in psychology at the moment! Doesn’t happen that
often! These are interesting times!