http://www.guardian.co.uk/science/lost-worlds/2012/dec/01/dinosaurs-fossils?CMP=twt_gu
What peer review means for science
Dave Hone
Peer review is far from perfect, but the media presenting material that has not been through
review as accepted science is misleading
In my experience quite a lot of the misunderstandings over research, and the criticisms leveled at
science and scientists seems to stem from an ignorance of some relatively basic aspects of
scientific work. The Lost Worlds has not really been going long enough for me to have a great
handle on the audience I have here, but I do hope that it has a broad appeal and that a good
proportion of the readers are those without much of a science background. Even those that do
may not have a great concept on how many things are done – it wasn't until my PhD that I really
got a proper understanding on how scientific papers are published and the differences between
say reviewed papers and book chapters, and I don't think it was even mentioned while I was
working on my BSc. Naturally I can't say much about the vast majority of science degrees in the
U.K., but I wouldn't be surprised if there's a good number of graduates who, like me, were never
really introduced to aspects of science like getting papers published and applying for grants.
These issues and others are things I want to cover on here as I think that people having an idea
about them and how they work will give people a better appreciation of science and importantly,
provide a basis for avoiding bad science.
Let's take peer review. In the last few months there have been a couple of incidents where the
popular media have leaped on ideas about extinct reptiles and promoted them as, well if not quite
true, then certainly highly credible. Huge numbers of people would have seen IO9s coverage of
'vampire pterosaurs' and others would have heard the BBC's radio coverage of the 'aquatic
dinosaurs' idea (which was also covered in plenty of other places too). In both cases,
palaeontologists responded quickly and pointed out the huge holes in the proposals and lack of
scientific data supporting them (like here, here and here). However, quite how many people saw
those rebuttals is another question and I'm sure there's a good number of people out there who
saw these things presented as credible science and still labour under the misapprehension that
they are mainstream views and quite correct. The question is raised therefore as to why were
news organisations promoting such nonsense in the first place, and could they have known that
they were not credible?
The short answer is yes. While a quick call to an expert would have revealed the problem, the
fact that neither had been through peer review should have had alarm bells ringing.
Now I'm not pretending that peer review is brilliant, it has plenty of problems. Enough bad
papers get through and even the best researchers occasionally make major mistakes or miss those
of others. However, the fact is that it does at least provide a minimum amount of screening of
scientific research and hypotheses, and ideas that have not passed it should be treated with
greater suspicion than those that have. With the examples listed above, one idea was published in
a popular science magazine (i.e. one full of articles aimed at the public, rather than technical,
scientific publications) and the other was a formal conference abstract (though again, not
reviewed) and backed by a personal website. I suspect it might be obvious to even non-scientists
that neither sound like the right kind of venue to present (allegedly) important new research to
the scientific community.
So, what is peer review? Quite simply, when a researcher submits a manuscript to a technical
scientific journal, the editors will check the paper for basic competence and then send it onto a
number of expert referees (generally two, sometimes many more) to check the actual science.
These people are generally experts in the field in question (i.e. a paper on tyrannosaur teeth
might be sent to a tyrannosaur specialist and a dinosaur tooth specialist) and they submit reports
to the journal about the quality of the work. Is all the necessary information in there? Were the
right analyses done in the right way? Is the science repeatable? Is it backed by the right
evidence? Are the conclusions supported by the data? Is this original research with proper credit
to previous studies? and so on. Assuming the paper is considered suitable, these reports are
passed back to the author to make changes suggested by the referees or indeed to challenge them
if he disagrees. The paper may go through several rounds of review and ultimately be rejected if
it is not considered acceptable and might have to be binned or started again. Those who have
tried to get a paper published will know that a great many scientific referees are both thorough
and unforgiving and a manuscript is regularly improved by the feedback provided by referees.
This means that a paper that has got into a peer-reviewed journal has at least got past an expert
editor and a couple of referees. It might still be rubbish, but has gained the blessing of at least a
couple of people in the field who do know the subject well. Contrast this with an article
submitted to a popular magazine where the limit to the input of an editor might be to suggest
rewriting a few sentences. One can see why giving media coverage to the latter as likely being
accurate science on the par with reviewed research might be, at best, unwise.
I would hope that readers can also appreciate how things can be seen by the researchers when
their work is covered in the media. It can take months to get reviews back from journals and
there may be pages and pages of comments from each of two referees and the editor. Arguments
and modifications may go back and forth and additional rounds of review undergone or
additional data collected and analyses revised to get things up to scratch. Years can literally pass,
you go through debates and discussions with colleagues, refine your arguments and augment
your evidence and see your paper eventually published. A major news outlet picks it up and the
first thing you see at the bottom is a comment from some smart-assed troll asking how this
nonsense ever got published or suggesting some 'obvious' flaw in the analysis. One gets the
strong impression that they think researchers can simply write whatever they want in a paper,
send it to the journal and a few weeks later it appears in print, and thus one could write almost
anything and see it published.
That is of course exactly what can happen with something that is not peer-reviewed, hence it is
most important that journalists covering science recognize the difference between the two, as
indeed should the audience. But it's this lack of recognition for the process that means the
mainstream media can seriously push the idea that all dinosaurs were aquatic and stuck in the
water based on a scientific paper that was nothing of the kind.