Discussion about exploratory vs. confirmatory research. Definition: EJ o o Exploratory research uses the data twice basically. Confirmatory research uses the data to test hypotheses, almost only happens in medicine… Denny o Exploratory research is a non-complete set of procedures, building theory. o Confirmatory research can make use of pre-registration. Pre-registration is however the method, not the definition of confirmatory research. Almost no research in science is purely confirmatory right now. Educating current and future researchers about exploratory versus confirmatory research will greatly reduce the number of people presenting exploratory research as confirmatory. EJ: Definitely a good idea to educate on the difference! This should have its place early on in scientific education; it should be taught in the first year actually! It is at least good that there is a course like Good Science Bad Science now. Ideally, all research is clearly defined, and research with low power is not sensible. But it would be unfair of me to demand a high sample size in all research for example, because this is often not feasible. D: The difference is quite trivial. It’s hard to say how teaching this will turn out. There is always a continuum between the two types. ‘Science’ is a very big ‘field’ and psychology has a lot of different paradigms. There should be a “hardcore” of science. Genetics for instance, solved its own problem by only moving to publication when replication is confirmed. It’s important to handle data carefully. Who is in your opinion most responsible for the fact that people do exploratory analyses and report them as confirmatory?" D: it’s about the whole context in which this happens, it is a sort of network, everybody does it and everybody is responsible to some extent. Besides, it’s not true that exploratory research (or non-significant research!) won’t get published, people just don’t try it, they are stuck in their beliefs, stuck in the habits of the system. It’s basically bad science, about incredible theories, creating a wow effect. But if its truly good stuff and you find nothing, it will be published! EJ: Intermediate results are a problem: when there is no real clear effect but also no absence. If the evidence it’s not convincing/ambivalent you better try again. 0.06 is marginally significant; it shows a ‘trend’. Reviewers then tell us to do a second experiment. D: Peer reviewing sucks. Most of the time they are just inadequate at judging the quality of a paper and bring their own subjectiveness to the table. EJ: People are actually rooting for getting rid of journals. On the other hand, my papers get better through feedback. A solution to peer review could be to have no review at first. First put the paper on an open archive, sort of like pre-registering it and only after that let it be officially reviewed. D: Reviewing clouds science. Reviewers make the paper better even though the writer doesn’t understand what he is doing. So you are actually reading a paper that is a combination of the work and ideas of the original writer ànd the reviewer. There is a lot of nonsense out there. Do different fields in psychology differ in the exploratory vs confirmatory research debate? EJ: In social psychology they overestimate the extent to which they have theory --> we should have more theory --> but people who claim this are not able to make good theory. A good theory is a theory that extents to a model that makes predictions and can be directly tested. We should therefore move to theory that is more concrete either by using simulations or mathematical models. In certain areas, e.g. FMRI, it’s very expensive to collect data so it’s much more difficult to replicate (your own) research. Some fields also have smaller effect sizes, they might be more likely to “tweak” their data. Furthermore, within subject studies have higher power! it’s the between-subject experiments that are noisy --> within subject studies are more likely to be replicated. D: We don’t have a culture where theory construction is seen as a skill, a lot of people lack the ability of “theory construction”. In other scientific fields, e.g. physics, biology etc, theoretical science and practical science is clearly separated, people who construct theory don’t go into the field --> but this is not the case in psychology. How do we move psychology in the direction of theory construction? Psychology should have a broader focus. How do you know how to construct good theories if you don’t even know what good theories are? We should ask ourselves what the goal of science is? What should we try to achieve? Should a good researcher be measured by the amount of publications they have? EJ: The publication criterium is good for people who would otherwise be doing nothing. It’s difficult because you do need to have some sort of incentive. In the current publication system, at least the criteria are clear, that’s the advantage of publication criteria, so there’s no “bullying” from the alpha monkey. We need systems that can reward both individuals and groups. The current system does reward performance and merit… Would you recommend being a scientist? EJ/D: The biggest advantage of not being a scientist is that when you go home at 5pm you are done with you work – but not if you are a scientist. People work at night and in the weekend. Somebody said: in an ideal world, we should be able to be a good scientist with 40 hours work a week. but that’s not the case. Science is on your mind all the time --> it’s like running your own company. It looks like you have a lot of freedom, but at the same time you don’t. YOU are completely responsible for your own success. Maybe its comparable to a musician or artist --> it’s not a job, it’s a lifestyle. What is the most important skill for a psychologist/scientist? EJ/D: Thinking is the most important skill! Statistics is not the most important, it is just an indicator of a certain way of thinking, it is a symptom, not a cause of a good science. Another important skill is creative simulation! Thinking about what your theory does when it is concretely implemented. Another skill: writing! Enjoying writing is quite important. If you find it really difficult to express your ideas, it will be very difficult for you to survive in the current academic climate. You should have fun with what you’re doing! D: Basis of science: we are human, limited and need to acknowledge our mistakes and learn from them! You will fall, and then you have to get up. A crisis in psychology is a good thing! Being critical means we are in a state of improvement. Like Francis Bacon said: We are human and limited, acknowledge this and accept it and learn from our mistakes! EJ: We are witnessing a transitional stage in psychology at the moment! Doesn’t happen that often! These are interesting times!