sense about science
“I DON’T KNOW
WHAT TO
BELIEVE...”
Making sense of science stories
This leaflet is for people who follow debates about science and
medicine in the news. It explains how scientists present and judge research and
how you can ask questions of the scientific information presented to you.
sense about science
SUMMARY
• Science has a system for assessing the quality of research before
it is published. This system is called peer review.
• Peer review means that other scientific experts in the field
check research papers for validity, significance and originality –
and for clarity.
• Editors of scientific journals draw on a large pool of suitable
experts to scrutinise papers before deciding whether to publish
them.
• Many of the research claims you read in newspapers and
magazines, find on the internet, or hear on television and the
radio are not published in a peer-reviewed journal.
• Some of this research may turn out to be good but much of it is
flawed or incomplete. Many reported findings, such as claims
about “wonder cures” and “new dangers”, never come to anything.
• Unpublished research is no help to anyone. Scientists can’t repeat
or use it and as a society we can’t base decisions about our public
safety – or our family’s health for example – on work that has a
high chance of being flawed.
• So, no matter how exciting or compelling new scientific or medical
research is, you must always ask…
Is it peer reviewed? If not, why not?
If it is peer reviewed, you can look for more information on what
other scientists say about it, the size and approach of the study and
whether it is part of a body of evidence pointing towards the same
conclusions.
sense about science
HOW SHOULD YOU MAKE SENSE OF SCIENCE STORIES?
Every day we are bombarded with information about science from newspapers, radio
and television programmes and the internet. Making sense of it all can be very
difficult. What should be taken seriously? Which are ‘scares’? Sometimes scientists
are reported as saying conflicting things. How do we know what to believe?
There is a system used by scientists to decide which research results should be
published in a scientific journal. This system, called peer review, subjects scientific
research papers to independent scrutiny by other qualified scientific experts (peers)
before they are made public.
Peer review can help you make sense of science stories as it tells you that the
research has passed the scrutiny of other scientists and is considered valid,
significant and original.
Peer review means that statements made by scientists in scientific journals are
critically different from other kinds of statements or claims, such as those made by
politicians, newspaper columnists or campaign groups. Science is therefore more than
just another opinion.
A SHORT EXPLANATION OF PEER REVIEW
When a researcher, or team of researchers, finishes a stage of work, they usually write
a paper presenting their methods, findings and conclusions. They then send the paper
to a scientific journal to be considered for publication.
If the journal’s editor thinks it is suitable for their journal they send the paper
to other scientists who research and publish in the same field asking them to:
• Comment on its validity – are the research results credible;
are the design and methodology appropriate?
• Judge the significance - is it an important finding?
• Determine its originality - are the results new?
Does the paper refer properly to work done by others?
• Give an opinion as to whether the paper should be published,
improved or rejected (usually to be submitted elsewhere).
This process is called peer review. The scientists (peers)
assessing the papers are called referees or reviewers.
Scientists never draw firm conclusions from just one paper or set of results.
They consider the contribution it makes in the context of other work and their
own experience. It usually takes more than one research paper for results to be
seen as good evidence or accepted as a public truth.
sense about science
THE SCIENCE PUBLISHING SCENE
For scientific knowledge to progress scientists need to share their research findings
with other scientists. The main way they do this is by publishing their research in
scientific journals – periodical publications intended to further the development of
science by reporting new research.
Journal editors receive many more papers than they can publish, so they use a twostep selection process. First, they consider whether the paper is a ‘fit’ for their
journals. For example, some journals only publish research papers that are
groundbreaking; others only publish research in a specific area, such as microbiology.
If a journal editor decides that a paper is right for their journal, they send it for peer
review to check whether the research findings are valid, significant and original.
A NOTE ON JOURNAL
FUNDING AND
AVAILABILITY
Most journals receive their income
from subscriptions and some from
organisational subsidies, conference
organising and advertising.
Many are available on the internet
and, increasingly, make their
online content free after a certain
period, usually one year.
There are alternative journal
funding models, such as scientists
paying the costs of reviewing and
publishing their articles so that they
can be made freely available. Less
than 1% of papers are published
this way.
Did you know?
There are around 21,000 scholarly
and scientific journals that use the
peer-review system. A high proportion
of these are scientific, technical or
medical journals, publishing over
1 million research papers each year.
sense about science
Publishing in a journal is an integral part
of being a scientist.
It:
• Connects like-minded individuals and
tells them about new research.
A published paper is read by scientists
all around the world.
• Is a permanent record of what has
been discovered, when and by which
scientists – like a court register for
science.
• Helps scientists to promote their work
and gain recognition from funders and
other institutions.
• Shows the quality of the scientist’s
work: other experts have rated it as
valid, significant and original.
By the way...
Peer review of research proposals
Peer review is also used to assess
scientists’ applications for research
funds. Funding bodies, such as medical
research charities, seek expert advice
on a scientist’s proposal before
agreeing to pay for it. Peer review in
this instance is used to judge which
applications are the best science and
have the potential to help the
organisation achieve its objectives.
HOW CAN YOU TELL
WHETHER REPORTED
RESULTS HAVE BEEN
PEER REVIEWED?
It can be difficult!
The full reference to peer-reviewed
papers is likely to look like this:
Fellers J H and Fellers G M (1976)
Tool use in a social insect and its
implications for competitive
interactions. Science, 192, 70-72.
…or this:
Hedenfalk I, Duggan D, Chen Y, et al.
Gene-expression profiles in
hereditary breast cancer.
N Engl J Med, 2001; 344: 539-48.
A few u
nscr up
people
ulous
use this
style o
and in
n webs
ar ticles
ites
to cite
not pee
w
ork tha
r review
t is
ed. But
for tuna
this is
t
e
ly
rare.
You are most likely to hear about
new research from the daily news
media, where there is not space or
interest in full references. Good
journalists usually indicate whether
research has been published and
mention the name of the journal.
sense about science
You can also look for longer reports
of the same research in other
newspapers, or popular science
magazines, many of which are
online, to find out whether research
is published and where.
This also helps for clarifying whether
the reported claims are a true
reflection of the findings in the
research paper.
Resear
ch pap
at scie
ers pre
ntific c
sented
o
n
fe
rences
begun
a proce
have o
ften
ss of p
are usu
eer rev
ally stil
ie
w
but
l unpub
lished
prelimin
and
ar y.
The more we ask, ‘is it peer
reviewed?’ the more obliged
reporters will be to include this
information.
There is no definitive list of peerreviewed journals but you can look
up the names of selected peerreviewed journals online at the
science news service EurekAlert!
(www.eurekalert.org/links.php?jrnl=A)
Sources of further help with
ascertaining the status of
research are listed at the end
of this leaflet.
SO SCIENTISTS USE PEER
REVIEW, SO WHAT?
When research findings have been peer
reviewed and published in a scientific
journal, this indicates that they are
sufficiently valid, significant and original
to merit the attention of other scientists.
Peer review is an essential dividing line
for judging what is scientific and what is
speculation and opinion. Most scientists
make a careful distinction between their
peer-reviewed findings and their more
general opinions.
Sounds good, but what happens
next?
Publication of a peer-reviewed paper is
just the first step: findings, and theories
about them, must go on to be re-tested
and judged against other work in the
same area. Some papers’ conclusions will
be disputed or further research will show
that they need to be revised as more data
are gathered.
Just as a washing machine has a
quality kite-mark, so peer review is a
kind of quality mark for science. It
tells you that the research has been
conducted and presented to a
standard that other scientists accept.
sense about science
CHALLENGES FOR PEER REVIEW
Why can’t there just be a checklist of
scientific validity?
Assessing scientific papers cannot be done in the
same way as giving a car an M.O.T. or marking a
maths test. New research usually has its own
unique features, which are difficult to predict with
a check list and which require expert judgement about
their validity, significance and originality.
Does peer review detect fraud and misconduct?
Peer review is not a fraud detection system. Referees are likely to detect
some wrongdoing, such as copying someone else’s research or misrepresenting data,
because they care about their subject. They know what research has been conducted
already and the kinds of results that are likely. However, if someone deliberately sets
out to falsify data, there is sometimes no way of knowing this until the paper is
published and others in the scientific community scrutinise and try to repeat the work.
Is ‘maverick’ science rejected through peer review?
Sometimes people worry that new ideas won’t be understood by other scientists
(although this is also an excuse given when researchers don’t want to submit to the
scrutiny of their peers). It is true that referees can be cautious about unusual findings;
and important insights can initially be overlooked. But if someone has been
exceptionally clever, other scientists are most likely to recognise it and to distinguish it
from flawed or inflated claims. Journal editors like novel ideas and scientific publishing
has brought thousands of important discoveries to light.
Does the peer-review process slow down advances in scientific and
medical knowledge?
In our world of instant communication and 24-hour news, a deliberative process like
peer review can seem frustratingly slow. Electronic communication has improved it, but
good assessment of research does take time. Sometimes people justify the promotion
of unpublished findings by saying they are ‘too important to wait’. But, although some
papers take months to review and improve, if there is a major breakthrough the
process can be completed in weeks. Furthermore, if the findings are very important –
e.g. they concern public health – then it is all the more necessary to check them
through peer review.
sense
senseabout
aboutscience
science
sense
about
science
sense
about
science
senseabout
aboutscience
science
sense
SOURCES
SOURCESFOR
FORFURTHER
FURTHERINFORMATION
INFORMATION
Sense
Sense
About
About
Science:
Science:
To To
findfind
outout
more
more
about
about
peer
peer
review
review
youyou
cancan
visit
visit
thethe
Sense
Sense
About
About
Science
Science
website
website
where
where
there
there
is aissection
a section
dedicated
dedicated
to it.
to The
it. The
section
section
includes
includes
freefree
downloads
downloads
of Sense
of Sense
About
About
Science’s
Science’s
longer
longer
report:
report:
Peer
Peer
Review
Review
andand
thethe
Acceptance
Acceptance
of New
of New
SOURCES
FOR
FURTHER
INFORMATION
SOURCES
FOR
FURTHER
INFORMATION
SOURCES
FOR
FURTHER
INFORMATION
Scientific
Scientific
Ideas
Ideas
(2004),
(2004),
electronic
electronic
versions
versions
of
the
of
the
leaflet
leaflet
and
and
additional
additional
educational
educational
SOURCES
FOR
FURTHER
INFORMATION
Sense
About
Science:
out
more
about
peer
review
youcan
canvisit
visitthe
theSense
SenseAbout
About
Sense
About
Science:
ToTofind
out
more
about
About
Sense
About
Science:
Tofind
find
out
more
aboutpeer
peerreview
reviewyou
you
can
visit
the
Sense
resources.
resources.
To
To
request
request
further
copies
ofdedicated
the
of
the
leaflet
leaflet
please
please
email:
Science
website
where
there
section
dedicated
it.The
The
section
includes
free
Science
website
where
there
isismore
aisasection
toto
section
includes
Sense
About
Science:
To further
find
out
about
peer
review
you
can
visitemail:
the
Sensefree
About
Science
website
where
there
acopies
section
dedicated
toit.
it.
The
section
includes
downloads
Sense
About
Science’s
longer
report:
Peer
Review
and
theAcceptance
Acceptance
New
publications@senseaboutscience.org
publications@senseaboutscience.org
or call:
or
call:
+44
+44
(0)
(0)
20
20
7478
7478
4380
4380
downloads
of
Sense
About
longer
report:
Peer
Review
and
the
Acceptance
of
downloads
ofofSense
About
Science’s
report:
Review
and
the
ofofNew
New
Science
website
where
there
is
aScience’s
sectionlonger
dedicated
to Peer
it.
The
section
includes
free
Scientific
Ideas
(2004),
electronic
versions
the
leaflet
andadditional
additional
educational
Scientific
Ideas
(2004),
electronic
versions
ofthe
the
and
additional
educational
Scientific
(2004),
electronic
versions
ofof
leaflet
and
educational
downloads
ofIdeas
Sense
About
Science’s
longer
report:
Peer
Review
and
the Acceptance
of New
www.senseaboutscience.org/peerreview
www.senseaboutscience.org/peerreview
s
resources.
request
further
copies
the
leaflet
please
email:
resources.
To
request
further
copies
the
leaflet
please
email:
resources.
ToTo
request
further
copies
ofofof
the
leaflet
please
Scientific
Ideas
(2004),
electronic
versions
of
the
leaflet
andemail:
additional
educational
Association
Association
of Medical
of Medical
Research
Research
Charities:
Charities:
The
The
AMRC
AMRC
has
has
a page,
a page,
for for
medical
medical
research
research
publications@senseaboutscience.org
call:
+44
(0)
7478
4380
publications@senseaboutscience.org
or
call:
+44
20
7478
4380
publications@senseaboutscience.org
oror
call:
+44
(0)
20
7478
4380
resources.
To request further copies of the
leaflet
please
email:
charities,
charities,
on on
thethe
peer
peer
review
review
of research
of
grant
grant
applications:
applications:
www.senseaboutscience.org/peerreview
www.senseaboutscience.org/peerreview
www.senseaboutscience.org/peerreview
publications@senseaboutscience.org
or research
call: +44
(0)
20
7478
4380
www.amrc.org.uk/temp/Aboutsppeerspreview.doc
www.amrc.org.uk/temp/Aboutsppeerspreview.doc
Association
of
Medical
Research
Charities:
TheAMRC
has
page,
for
medical
www.senseaboutscience.org/peerreview
Association
Medical
Research
Charities:
The
AMRChas
hasaaapage,
page,for
formedical
medicalresearch
research
Association
ofofMedical
Research
Charities:
The
research
““
charities,
on
the
peer
review
of
research
grant
applications:
charities,
onMedical
the
peer
review
research
grant
applications:
Association
ofon
Research
The
AMRC
has
for board
medical
research
charities,
on
the
peer
review
ofofCharities:
research
grant
applications:
Committee
Committee
on
Publication
Publication
Ethics:
Ethics:
COPE
COPE
provides
provides
a asounding
apage,
sounding
board
for
for
journal
journal
editors
editors
www.amrc.org.uk/temp/Aboutsppeerspreview.doc
www.amrc.org.uk/temp/Aboutsppeerspreview.doc
charities,
on
the
peer
review
of
research
grant
applications:
www.amrc.org.uk/temp/Aboutsppeerspreview.doc
struggling
struggling
with
with
howhow
to deal
to deal
with
with
breaches
breaches
in research
in research
andand
publication
publication
ethics:
ethics:
Committee
on
PublicationEthics:
Ethics:COPE
COPE provides
provides a sounding
sounding board
for
journal
editors
www.amrc.org.uk/temp/Aboutsppeerspreview.doc
Committee
Publication
boardfor
forjournal
journaleditors
editors
Committee
ononPublication
Ethics: COPE
provides a asounding
board
www.publicationethics.org.uk
www.publicationethics.org.uk
struggling with how to deal with breaches in research and publication ethics:
struggling
with
howtotodeal
deal
withbreaches
breaches
research
andpublication
publication
ethics:
Committee
on
Publication
Ethics:
COPE
provides
a sounding
board
for journal
editors
struggling
with
how
with
ininresearch
and
ethics:
ta,
ed
s
rk.
www.publicationethics.org.uk
The
The
National
National
Electronic
Electronic
Library
Library
for for
Health:
Health:
TheThe
NELH
NELH
hashas
a ‘Hitting
a ‘Hitting
thethe
Headlines’
Headlines’
archive,
archive,
www.publicationethics.org.uk
struggling
with how to deal with breaches in research and publication ethics:
www.publicationethics.org.uk
The
National
Electronic
Library
for
Health:
The
NELH
has
a
‘Hitting
the
Headlines’
archive,
which
which
looks
looks
at
medical
at
medical
news
news
stories
stories
and
and
provides
provides
the
the
research
research
evidence
evidence
on
on
which
which
they
they
areare
www.publicationethics.org.uk
TheNational
NationalElectronic
ElectronicLibrary
Libraryfor
forHealth:
Health:The
TheNELH
NELHhas
hasa a‘Hitting
‘Hittingthe
the Headlines’archive,
archive,
The
which
looks at medical
news stories
and provides
the research
evidenceHeadlines’
on which they are
based:
based:
www.library.nhs.uk/rss/newsAndRssArchive.aspx?storyCatagory=1
www.library.nhs.uk/rss/newsAndRssArchive.aspx?storyCatagory=1
which
looks
medical
newsstories
stories
andThe
provides
theresearch
research
evidence
which
theyare
are
The
National
Electronic
Library
for
Health:
NELHthe
has
a
‘Hittingevidence
the Headlines’
archive,
which
looks
atatmedical
news
and
provides
ononwhich
they
based: www.library.nhs.uk/rss/newsAndRssArchive.aspx?storyCatagory=1
based:
www.library.nhs.uk/rss/newsAndRssArchive.aspx?storyCatagory=1
which
looks
at Media
medical
news
stories
and
provides
the
research
evidence
onPeer
which
they in
areainNutshell,
based:
www.library.nhs.uk/rss/newsAndRssArchive.aspx?storyCatagory=1
The
The
Science
Science
Media
Centre:
Centre:
TheThe
SMC
SMC
hashas
published
published
a leaflet,
a leaflet,
Peer
Review
Review
a Nutshell,
a a
The Science Media Centre: The SMC has published a leaflet, Peer Review in a Nutshell, a
based:
www.library.nhs.uk/rss/newsAndRssArchive.aspx?storyCatagory=1
The
Science
Media
Centre:
The
SMC
has
published
a
leaflet,
Peer
Review
in
a
Nutshell,
guide
guide
for
for
scientists
scientists
preparing
preparing
for
for
a
news
a
news
interview:
interview:
The guide
Science
Media Centre:
The for
SMC
has published
for scientists
preparing
a news
interview: a leaflet, Peer Review in a Nutshell, a a
guide
forscientists
scientists
preparing
fora anews
news
interview:
The
Science
Media
Centre:
The for
SMC
has
published
a leaflet, Peer Review in a Nutshell, a
guide
for
preparing
interview:
www.sciencemediacentre.org/peer_review.htm
www.sciencemediacentre.org/peer_review.htm
www.sciencemediacentre.org/peer_review.htm
www.sciencemediacentre.org/peer_review.htm
guide
for scientists preparing for a news interview:
www.sciencemediacentre.org/peer_review.htm
www.sciencemediacentre.org/peer_review.htm
e
gs;
ACKNOWLEDGEMENTS
ACKNOWLEDGEMENTS
ACKNOWLEDGEMENTS
ACKNOWLEDGEMENTS
ACKNOWLEDGEMENTS
This leaflet has been produced and distributed with sponsorship and help from:
ACKNOWLEDGEMENTS
This
This
leaflet
leaflet
has
has
been
been
produced
produced
and
and
distributed
distributed
with
sponsorship
sponsorship
and
and
help
help
from:
from:
This
leaflet
has
been
produced
and
distributed
withwith
sponsorship
andhelp
help
from:
This
leaflet
has
been
produced
and
distributed
with
sponsorship
and
from:
This leaflet has been produced and distributed with sponsorship and help from:
h it
ing
medi
me
med
h
medic
ho
nd
but
ion
me
–
Sense About Science is grateful for the input of the sponsors, the many organisations (in
particular Cancer Research UK, Asthma UK, Migraine Trust and Action Medical Research),
SenseAbout
AboutScience
Scienceisisgrateful
grateful forthe
theinput
inputofofthe
the sponsors,the
the manyorganisations
organisations(in(in
Sense
parliamentarians,
government for
officials,
educationalsponsors,
organisations,many
teachers, school
particular
Cancer
Research
UK,
Asthma
UK,
Migraine
Trust
and
Action
Medical
Research),
Sense
About
Science
is grateful
forAsthma
the
input
of
the
sponsors,
the
many
organisations
(in
particular
Cancer
Research
UK,
Migraine
Trust
and
Action
Medical
Research),
Sense
About
Science
isUK,
grateful
for
the
input
of
sponsors,
the
many
organisations
Sense
About
Science
is grateful
for
the
input
of
sponsors,
the
many
organisations
(in (in
students,
doctors,
pharmacists,
science
bodies
andthe
thethe
many
others,
who
kindly
contributed
parliamentarians,
government
officials,
educational
organisations,
teachers,
school
particular
Research
UK, UK,
Asthma
UK,
Migraine
Trust
and
Action
Medical
Research),
parliamentarians,
government
officials,
educational
organisations,
teachers,
school
theirCancer
timeCancer
and
ideas.
Responsibility
for
the
content
rests
fully
withand
Sense
About
Science.
particular
particular
Cancer
Research
Research
UK,
Asthma
Asthma
UK,
UK,
Migraine
Migraine
Trust
Trust
and
Action
Action
Medical
Medical
Research),
Research),
students,doctors,
doctors,
pharmacists,
science
bodiesand
andthe
themany
manyothers,
others,who
who
kindlycontributed
contributed
parliamentarians,
government
officials,
educational
organisations,
teachers,
school
students,
pharmacists,
science
bodies
kindly
The Lancet covergovernment
reprinted
from The
Lancet
, Vol. 366,
No. 9487,
27
August 2005, teachers,
withteachers,
permission
parliamentarians,
government
officials,
educational
organisations,
school
parliamentarians,
officials,
organisations,
school
theirtime
time
andideas.
ideas.
Responsibility
for
theeducational
content
rests
fully
withSense
Sense
About
Science.
students,
doctors,
pharmacists,
science
bodies
and the
many
others,
who
kindly
contributed
their
and
Responsibility
for
the
content
rests
fully
with
About
Science.
from
Elsevier.
students,
doctors,
pharmacists,
science
bodies
and
the
many
others,
who
kindly
contributed
students,
doctors,
pharmacists,
science
bodies
and
the
many
others,
who
kindly
contributed
their
time
andcover
ideas.
Responsibility
for
the
rests
fully27
with
Sense
About
Science.
The
Lancet
cover
reprinted
fromThe
TheLancet
Lancet
,content
Vol.366,
366,No.
No.9487,
9487,
27August
August
2005,
with
permission
The
Lancet
reprinted
from
, Vol.
2005,
with
permission
Science cover reprinted with permission from AAAS.
from ©Elsevier.
Sense About Science 2005. Reprinted 2010.
Science
cover
reprinted
with
permission
from
from
Elsevier.
The
The
Lancet
Lancet
cover
cover
reprinted
reprinted
from
TheThe
Lancet
Lancet
,AAAS.
Vol.
, Vol.
366,
366,
No.
No.
9487,
9487,
27 27
August
August
2005,
2005,
withwith
permission
permission
Science
cover
reprinted
with
permission
from
AAAS.
Sense
About
Science,
25from
Shaftesbury
Avenue,
London
W1D
7EG
Science
cover
reprinted
with
permission
from2010.
AAAS.
©Sense
Sense
About
Science
2005.
Reprinted
2010.
Registered
charity
number
1101114
from
from
Elsevier.
Elsevier.
©
About
Science
2005.
Reprinted
their
time
time
andand
ideas.
ideas.
Responsibility
Responsibility
for for
thethe
content
content
rests
rests
fullyfully
with
with
Sense
Sense
About
About
Science.
Science.
from
Elsevier.
Thetheir
Lancet
cover reprinted from The Lancet, Vol. 366, No. 9487, 27 August 2005, with permission
© Sense
About
Science
2005.
Reprinted
2010.
Sense
About
Science,
Shaftesbury
Avenue,
London
W1D7EG
7EG
Science
Science
cover
cover
reprinted
reprinted
with
with
permission
permission
from
from
AAAS.
AAAS.
Sense
About
Science,
2525
Shaftesbury
Avenue,
London
W1D
Registered
charitynumber
number
1101114
Sense
About Science,
25
Shaftesbury
Avenue, London W1D 7EG
Registered
charity
1101114
© Sense
© Sense
About
About
Science
Science
2005.
2005.
Reprinted
Reprinted
2010.
2010.
Registered charity number 1101114
Sense
Sense
About
About
Science,
Science,
25 25
Shaftesbury
Shaftesbury
Avenue,
Avenue,
London
London
W1D
W1D
7EG7EG
Registered
Registered
charity
charity
number
number
1101114
1101114
M
M