Competências Básicas de Investigação
Científica e de Publicação
Lecture 2: Hypotheses, reproducibility and
experimental design
9 September 2013
Ganesha Associates CC BY 3.0
1
The scientific process involves making
models of how things work
• These evolving models are described in the
scientific literature
• Sometimes the models are wrong, often they are
incomplete
• Scientific progress is driven by the
communication and publication of the results of
new research, and the reinterpretation of older
work
• The tool which makes all of this possible is the
hypothesis
9 September 2013
Ganesha Associates CC BY 3.0
2
The scientific process involves making
models of how things work
9 September 2013
Ganesha Associates CC BY 3.0
3
9 September 2013
Ganesha Associates CC BY 3.0
4
9 September 2013
Ganesha Associates CC BY 3.0
5
9 September 2013
Ganesha Associates CC BY 3.0
6
A visual abstract from Cell
9 September 2013
Ganesha Associates CC BY 3.0
7
Experimental vs. Observational studies
No modification of experimental variables
Useful to discover trends and associations
Cannot directly be used to infer causality
Compare responses different treatments
Designed to avoid misleading results
e.g. randomisation
Can be used to infer cause and effect
9 September 2013
Ganesha Associates CC BY 3.0
8
Main learning points
• Student projects fall into three categories
– No hypothesis, i.e. observational
– Weak hypothesis
– Strong hypothesis
• The work will be published in a
– National journal
– Low impact factor journal
– High impact factor journal
• Starting with strong hypothesis improves your
chances of getting published in a good journal
9 September 2013
Ganesha Associates CC BY 3.0
9
9 September 2013
Ganesha Associates CC BY 3.0
10
What is a strong hypothesis ?
• A strong hypothesis is based on a series of
premises – things that are already known with
some certainty
• Each premise must be supported by
references back to the (international) primary
literature
• So a strong hypothesis will be backed by
references to recent papers in high quality
journals
9 September 2013
Ganesha Associates CC BY 3.0
11
9 September 2013
Ganesha Associates CC BY 3.0
12
Why did the Titanic sink?
It was Captain Smith's fault
This was Captain E. J. Smith's retirement trip. All he had to do was get to New York in record time. Captain E. J. Smith said years
before the Titanic's voyage, "I cannot imagine any condition which would cause a ship to founder. Modern shipbuilding has gone
beyond that." Captain Smith ignored seven iceberg warnings from his crew and other ships. If he had called for the ship to slow
down then maybe the Titanic disaster would not have happened.
It was the shipbuilder's fault
About three million rivets were used to hold the sections of the Titanic together. Some rivets have been recovered from the wreck
and analysed. The findings show that they were made of sub-standard iron. When the ship hit the iceberg, the force of the impact
caused the heads of the rivets to break and the sections of the Titanic to come apart. If good quality iron rivets had been used the
sections may have stayed together and the ship may not have sunk.
It was Bruce Ismay's fault
Bruce Ismay was the Managing Director of the White Star Line and he was aboard the Titanic. Competition for Atlantic passengers
was fierce and the White Star Line wanted to show that they could make a six-day crossing. To meet this schedule the Titanic
could not afford to slow down. It is believed that Ismay put pressure on Captain Smith to maintain the speed of the ship.
It was Thomas Andrews' fault
The belief that the ship was unsinkable was, in part, due to the fact that the Titanic had sixteen watertight compartments.
However, the compartments did not reach as high as they should have done. The White Star Line did not want them to go all the
way up because this would have reduced living space in first class. If Mr Andrews, the ship's architect, had insisted on making
them the correct height then maybe the Titanic would not have sunk.
It was Captain Lord's Fault
The final iceberg warning sent to Titanic was from the Californian. Captained by Stanley Lord, she had stopped for the night about
19 miles north of Titanic. At around 11.15, Californian's radio operator turned off the radio and went to bed. Sometime after
midnight the crew on watch reported seeing rockets being fired into the sky from a big liner. Captain Lord was informed but it was
concluded that the ship was having a party. No action was taken by the Californian. If the Californian had turned on the radio she
would have heard the distress messages from Titanic and would have been able to reach the ship in time to save all passengers.
9 September 2013
Ganesha Associates CC BY 3.0
13
Coin-tossing - an example
• I wonder how many heads or tails I will get if I toss
this coin 100 times
– No model
• The frequency distribution of heads and tails will be
approximated by a binomial distribution with n=100
and p=0.5
– Simple model, based on symmetry
• A detailed analysis of the dynamics reveals that the
probability of a head is 0.51
– Complex model, based on asymmetry, aerodynamics, etc
9 September 2013
Ganesha Associates CC BY 3.0
14
Coin-tossing – impact on CV
1. None, or possibly negative
2. R. A. Fisher and others did perform this experiment in the early
days of biological statistics, before the advent of computers, as a proof
that the binomial distribution tended towards a normal one at high
levels of n.
Interestingly they all found that the probability of a head p was usually
slightly higher than 0.5, but this difference was ignored.
3. Persi Diacusis, Susan Holmes and Richard Montgomery (Stanford,
2004) publish a paper on the ‘Dynamical bias in the coin toss’ proving
that the lack of total symmetry in a coin means that the probability of
a head will always be slightly greater than 0.5.
9 September 2013
Ganesha Associates CC BY 3.0
15
Coin tossing - relevance
• I think that there will be an association (+ or -) between mutations
in gene x and susceptibility to disease y
– No causal basis for a relationship given
• I predict that mutations in gene x will increase susceptibility to
disease y because patients with disease y often have low levels of
gene product x.
– Built-in control, patients with normal levels of the gene product should not
have the disease.
• I predict that chemically non-neutral mutations in gene x will
increase susceptibility to disease y in patients with low levels of
gene product x.
– Second level of control – neutral mutations should be asymptomatic
9 September 2013
Ganesha Associates CC BY 3.0
16
Coin-tossing – moral of the story
• With a strong hypothesis, you:
– Avoid following leads which go nowhere – false
positives, fail early
– Avoid ignoring unexpected observations that are
of high interest – false negatives
– May need to do less work !
– Will get published in better journals !
9 September 2013
Ganesha Associates CC BY 3.0
17
Testing a hypothesis - 1
• The first stage of any scientific analysis is to define
the null hypothesis that is to be tested.
• This makes the prediction that our chosen variable
has no impact on the outcome we are measuring.
• As an example, the null hypothesis could be that
“there is no difference in rainfall levels between
urban and adjacent rural areas”.
• The null hypothesis is denoted symbolically as H0
9 September 2013
Ganesha Associates CC BY 3.0
18
Testing a hypothesis - 2
• It is also necessary to state the alternative hypothesis
(H1) – that there is an increase in precipitation levels
in urban areas relative to adjacent rural areas
because of the heating differences of the two surface
types (the urban area heats up more and has
increased convective uplift).
• Notice, it is a lot easier to specify the H0 version. H1
requires an understanding of convection,
thermodynamics and so forth.
9 September 2013
Ganesha Associates CC BY 3.0
19
Testing a hypothesis - 3
• The H0 hypothesis protects us from trying to
explain results that have no significance.
• The H1 hypothesis challenges us to make sure
we have thought of all of the other factors
that could cause the effects we observe and
excluded them by appropriate experimental
controls or via additional experiments.
9 September 2013
Ganesha Associates CC BY 3.0
20
Testing a hypothesis - 4
• You can often spot problems with project design by
analyzing the Introduction to a research proposal.
• If you edit out repetitive statements and non-essential
background information, you should be left with a logical
sequence of statements which leads to a clear H1
hypothesis.
• Unlike H0 hypotheses – there is no treatment effect – you
have to have a mechanism in mind in order to formulate
the H1 hypothesis.
• Without this you have no logical structure to connect
your results back to a known process or mechanism.
9 September 2013
Ganesha Associates CC BY 3.0
21
Testing a hypothesis - 5
•
For example, this is the edited version of the introduction to a scientific paper. Other
than remove a couple of repetitious statements, I have made no changes.
1.
Ectomycorrhizal fungi (EMF) have a fundamental role in nutrient absorption of many plant species.
2. Tree species of ecological and economic relevance in reforestation programs depend on ectomycorrhizal
symbiosis, especially in soils contaminated by mining activities.
3. The ability of EMF to reduce the toxicity of heavy metal ions (e.g. copper) in their host plants is
accompanied by the decrease of metal concentrations in the aerial part of the plant.
4. Due to increased absorption of these metals by the EMF roots and the accumulation in the extra radical
mycelium, greater tolerance to such elements is achieved by the host.
5. Research has been conducted to determine the sensitivity of EMF to a variety of potentially toxic metals to
understand the diverse mechanisms through which the fungi may tolerate heavy metals.
6. Enzymatic activity is important for the mobilization and transference of soil nutrients through EM fungi
towards the host plant.
7. In this study we investigate the effects of copper and phosphorus concentrations on mycelial growth and
enzymatic activities of the EM fungi Pisolithus microcarpus, Chondrogaster angustisporus and Suillus sp. in two
growth experiments.
9 September 2013
Ganesha Associates CC BY 3.0
22
Testing a hypothesis - 6
• The logical discontinuity occurs between statements 6 and
7. Since copper is toxic and phosphorous a fertilizer, we are
almost certainly going to see effects. But what do they mean
?
• There is actually quite a large literature on copper toxicity in
plants. Several mechanisms have been identified and it would
have been possible by choosing the experimental conditions
carefully to distinguish which ones were operating under this
particular set of experimental conditions.
• Hence the introduction should have continued after point 6 to
develop reasons for a more selective approach. This would
form the basis for the missing H1 hypothesis.
9 September 2013
Ganesha Associates CC BY 3.0
23
Case study: Hummingbird territorial
behaviour
9 September 2013
Ganesha Associates CC BY 3.0
24
Hummingbird territorial behaviour
Most hummingbird species demonstrate strong territorial behavior
If a bluffing charge attack does not work, the resident
may engage the trespasser in a brief but intense physical battle
So why do hummingbirds defend territories ?
H0: Hummingbirds are randomly distributed in space and time.
9 September 2013
Ganesha Associates CC BY 3.0
25
Hummingbird territorial behaviour
H1
If territory = F(energy), then behavior not species-dependent
If territory = F(mating), then behavior should be species and sex dependent
If…
If…
9 September 2013
Ganesha Associates CC BY 3.0
26
Territorial behaviour in 1971
• Time, Energy, and Territoriality of the Anna Hummingbird
(Calypte anna) Science 173 (1971) 818-821.
• When territory quality decreases defenders may
switch to less expensive forms of defense because
the energy savings outweigh the loss of resources
• Augmented territorial defense during the breeding
season is made possible by increased feeding
efficiency due to the availability at this time of very
nectar-rich flowers.
• Individuals with large territories are more successful
reproductively.
9 September 2013
Ganesha Associates CC BY 3.0
27
Hummingbird territoriality since
• Digestive physiology is a determinant of foraging bout
frequency in hummingbirds. Nature. 1986 Mar 612;320(6057):62-3.
• Mitochondrial respiration in hummingbird flight
muscles. Proc Natl Acad Sci U S A. 1991 Jun 1;88(11):4870-3.
• Cloning and analysis of the gene encoding hummingbird
proinsulin. Gen Comp Endocrinol. 1993 Jul;91(1):25-30.
• Flight and size constraints: hovering performance of
large hummingbirds under maximal loading. J Exp Biol.
1997 Nov;200(Pt 21):2757-63.
9 September 2013
Ganesha Associates CC BY 3.0
28
Hummingbird territoriality since
• Hovering performance of hummingbirds in hyperoxic
gas mixtures. J Exp Biol. 2001 Jun;204(Pt 11):2021-7.
• Adipose energy stores, physical work, and the
metabolic syndrome: lessons from hummingbirds.
Nutr J. 2005 Dec 13;4:36.
• Neural specialization for hovering in hummingbirds:
hypertrophy of the pretectal nucleus Lentiformis
mesencephali. J Comp Neurol. 2007 Jan 10;500(2):211-21.
• Three-dimensional kinematics of hummingbird flight.
J Exp Biol. 2007 Jul;210(Pt 13):2368-82.
9 September 2013
Ganesha Associates CC BY 3.0
29
Case Study One
– Observations
• There is evidence that the central nervous system can
influence bone metabolism
• Neurotransmitter (serotonin) receptors are found in bone
• Drugs that inhibit neurotransmitter release can affect bone
growth in humans
– Hypothesis
• If serotonin plays a role in regulating bone growth, then
reducing serotonin release in bone will slow growth
– Experiment
• Inject mice with drugs (SSRI’s) that reduce serotonin release
and measure impact on growth of femur over time
– Result
• Small but significant reduction in bone growth observed
9 September 2013
Ganesha Associates CC BY 3.0
32
Case Study One
– Observations
• There is evidence that the central nervous system can
influence bone metabolism
• Neurotransmitter (serotonin) receptors are found in bone
• Drugs that inhibit neurotransmitter release can affect
bone growth in humans
– Hypothesis
• If serotonin plays a role in regulating bone growth, then
reducing serotonin release in bone will slow growth
– Experiment
• Inject mice with drugs (SSRI’s) that reduce serotonin
release and measure impact on growth of femur over time
– Result
• Small but significant reduction in bone growth observed
– Referees comments
• Is this a direct or an indirect effect ?
9 September 2013
Ganesha Associates CC BY 3.0
33
Case Study Two
– Observations
• A plant extract (E) is used to treat diabetes
• Prenatal malnutrition may cause diabetes
– Hypothesis
• If E plays a role in regulating blood sugar levels then the
responses to E will be affected by dietary history
– Experiment
• Measure acute and chronic effects of E on glucose and
insulin levels in normal and prenatal malnourished rats
fed either on a standard or high glucose diet
– Result
• E does appear to reduce blood sugar levels but with no
significant differences between the experimental
groups of animals
9 September 2013
Ganesha Associates CC BY 3.0
34
Case Study Two
– Observations
• A plant extract (E) is used to treat diabetes
• Prenatal malnutrition may cause diabetes
– Hypothesis
• If E plays a role in regulating blood sugar levels then the responses to
E will be affected by dietary history
– Experiment
• Measure acute and chronic effects of E on glucose and insulin levels
in normal and prenatal malnourished rats fed either on a standard or
high glucose diet
– Result
• E does appear to reduce blood sugar levels but with no significant
differences between the experimental groups of animals
– Referees comments
• Statistical complexity of experimental design prevents any clear
conclusions being drawn about mechanism
9 September 2013
Ganesha Associates CC BY 3.0
35
Case Study Three
– Observation
• Fungal and other microbial symbionts can improve
nutrient uptake by plant roots
– Hypothesis
• Soil microbial diversity is adversely affected by
conventional farming techniques
– Experiment
• Compare microbial diversity associated with root soil in
vines cultivated using either conventional or organic
techniques
– Result
• Diversity and metabolic vigor higher in soils associated
with the roots of organically cultivated plants
9 September 2013
Ganesha Associates CC BY 3.0
36
Case Study Three
– Observation
• Fungal and other microbial symbionts can improve nutrient uptake
by plant roots
– Hypothesis
• Soil microbial diversity is adversely affected by conventional farming
techniques
– Experiment
• Compare microbial diversity associated with root soil in vines
cultivated using either conventional or organic techniques
– Result
• Diversity and metabolic vigor higher in soils associated with the roots
of organically cultivated plants
– Referees comments
• What was the control for this experiment ? Maybe the differences in
the two soil areas has nothing to do with agricultural methods.
9 September 2013
Ganesha Associates CC BY 3.0
37
Case Study Four
– Observation
• A patient suffers from an unusually aggressive form of
retinoblastoma
– Premise
• Patients who develop retinoblastoma also have a
particular type of genetic defect which can be identified
cytogenetically
– Hypothesis
• This patient has an unusual cytogenetic pattern which
could be useful diagnostically
– Result
• The patient does indeed have a unique cytogenetic
profile, but there are many differences from the
pattern normally seen in retinoblastoma cases
9 September 2013
Ganesha Associates CC BY 3.0
38
– Observation
Case Study Four
• A patient suffers from an unusually aggressive form of
retinoblastoma
– Premise
• Patients who develop retinoblastoma also have a particular
type of genetic defect which can be identified cytogenetically
– Hypothesis
• This patient has an unusual cytogenetic pattern which could be
useful diagnostically
– Result
• The patient does indeed have a unique cytogenetic profile
– Referees comments.
• In this patient there are many differences from the normal
pattern seen in retinoblastoma cases. How can we chose
which differences are causal?
9 September 2013
Ganesha Associates CC BY 3.0
39
Case Study Five
– Observation
• Infection by certain human papilloma virus serotypes
can lead to cervical cancer. One study has shown that
the prevalence of HPV 31 and 33 is much higher in
Recife than elsewhere in Brasil
• Mutations to human P53 are often associated with the
more severe consequences of HPV infection
– Hypothesis
• There is a strong correlation between patients with
advanced HPV 31 and 33 disease and a new p53
mutation which is found mainly in the NE of Brasil
– Result
• Patients with HPV 31 and 33 infections have higher
levels of cervical cancer and the new P53 mutation
compared to other regions in Brazil.
9 September 2013
Ganesha Associates CC BY 3.0
40
– Observation
Case Study Five
• Infection by certain human papilloma virus serotypes can
lead to cervical cancer. One study has shown that the
prevalence of HPV 31 and 33 is much higher in Recife than
elsewhere in Brasil
• Mutations to human P53 are often associated with the more
severe consequences of HPV infection
– Hypothesis
• There is a strong correlation between patients with
advanced HPV 31 and 33 disease and a new p53 mutation
which is found mainly in the NE of Brasil
– Result
• Patients with HPV 31 and 33 infections have higher levels of
cervical cancer and the new P53 mutation compared to
other regions in Brazil.
– Referees comments
• Isn’t this what we would expect from the existing
epidemiological data?
9 September 2013
Ganesha Associates CC BY 3.0
41
Case study summary
• These five hypotheses failed
• Either because they were not based on a rigorously
researched mechanism of action
• Or because the experimental design was weak
• As a consequence, important variables were not controlled for
• And the results could not be explained within the context of
the model
• Moral: Hypotheses are often wrong first time around
• Moral: Strong hypotheses usually require less work to prove
than weak ones !
9 September 2013
Ganesha Associates CC BY 3.0
42
Hypothesis lecture learning points
• Good hypotheses build directly onto previous
work
• So they need to become technically more
sophisticated over time moving from the
general to the particular
• A given problem can be associated with a
number of very different hypotheses – your
experiments should include tests to exclude
these alternative explanations
9 September 2013
Ganesha Associates CC BY 3.0
43
Hypothesis lecture learning points
• Hypotheses can be weak (observational) or
strong (mechanism-based)
• For example, a hypothesis which predicts that
a tossed coin will end up ‘heads’ 50% of the
time is much weaker than one that can predict
the exact sequence of ‘heads’ and ‘tails’
• So hypothesis ‘quality’ is important
9 September 2013
Ganesha Associates CC BY 3.0
44
Break
9 September 2013
Ganesha Associates CC BY 3.0
45
Reproducibilty
• No research paper can ever be considered to be the final
word
• Replication and corroboration of research results is key to
the scientific process.
• In studying complex entities, such as animals, ecosystems
and so forth, the complexity of the system and of the
techniques can all too easily lead to results that seem
robust in the lab, and valid to editors and referees of
journals, but which do not stand the test of reproducibilty.
• Journals, research laboratories and institutions and funders
all have an interest in tackling issues of irreproducibility.
9 September 2013
Ganesha Associates CC BY 3.0
46
“Why Most Published Research Findings
are False”
• Original paper by John Ioannidis published in PLoS Medicine in 2005 led to
much debate.
– Small sample sizes, weak hypotheses and experimental design, and bias mean
that most positive findings are incorrect.
• Commentary by Steven Goodman, ibid, 2007.
– Agreement that many medical research findings are less definitive than
readers suspect
– P-values are widely misinterpreted.
– Bias of various forms is widespread
– Multiple approaches are needed to prevent the literature from being
systematically biased
– Need for more data on the prevalence of false claims.
• Sources:
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.002
0124 and http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1855693/
9 September 2013
Ganesha Associates CC BY 3.0
47
Case study: Amgen retrospective
preclinical study
• Fifty-three papers were identified as ‘landmark’
studies in preclinical cancer research.
• However, scientific findings were confirmed in only
6 (11%) cases.
• Source: www.nature.com/nature/journal/v483/n7391/full/483531a.html
9 September 2013
Ganesha Associates CC BY 3.0
48
Amgen retrospective preclinical study
• The limitations of preclinical cancer models have been widely reviewed and are
largely acknowledged by the field.
• They include:
– the use of small numbers of poorly characterized tumour cell lines that inadequately
recapitulate human disease
– an inability to capture the human tumour environment
– a poor appreciation of pharmacokinetics and pharmacodynamics
– the use of problematic endpoints and testing strategies
– in addition, preclinical testing rarely includes predictive biomarkers that, when advanced to
clinical trials, will help to distinguish those patients who are likely to benefit from a drug.
• Learning point: Unfortunately, much published research ignores these
problems
– Read “Confiabilidade em crise” and “Nature Special: Challenges in irreproducible research”.
9 September 2013
Ganesha Associates CC BY 3.0
49