Research and how to do it
Introductory lectures for final year students and fresh
graduate students in the Faculty of Engineering
AY 09/26
PART 5: THE BALONEY DETECTION KIT
A/Prof Michael Raghunath, MD PhD
Division of Bioengineering, Faculty of Engineering
Department of Biochemistry, Faculty of Medicine
Tissue Modulation
Laboratory Manual
The demon-haunted world by Carl Sagan
is intended to explain the scientific
method to laypersons, and to
encourage people to learn critical or
skeptical thinking. It explains
methods to help distinguish between
ideas that are considered valid
science, and ideas that can be
considered pseudoscience. Sagan
states that when new ideas are
offered for consideration, they
should be tested by means of
skeptical thinking, and should stand
up to rigorous questioning.
Skepticism
Audiatur et altera pars (the other party
should also be heard)
In general, skepticism is a doubting or questioning
attitude, often associated with a doubting or
questioning attitude toward religion. Specifically,
skepticism is a philosophical movement that
emphasizes that absolute knowledge is unattainable,
and therefore inquiry must be a process of doubting
in order to acquire approximate or relative certainty.
1
The baloney detection kit
• independent confirmation of the facts
• encourage substantive debate on evidence by
knowledgeable proponents of all points of view
• arguments from authorities carry little weight (or
better put, there are no authorities in science, only
experts)
• spin more than one hypothesis
The baloney detection kit
•
•
•
•
•
do not get overly attached to a hypothesis
Quantify (seek numerical quantity )
in a chain of argument, every link must work
apply Occam’s Razor
always ask whether a hypothesis can be falsified
Like: All swans are white. If only one black swan can
be found, this hypothesis is falsified.
The baloney detection kit
Additional issues are
• Conduct control experiments
• Check for confounding factors - separate the
variables.
2
Common fallacies of logic and rhetoric
•Ad hominem - attacking the arguer and not the argument.
•Argument from "authority".
•Argument from adverse consequences (putting pressure on
the decision maker by pointing out dire consequences of an
"unfavourable" decision).
•Appeal to ignorance (absence of evidence is not evidence of
absence). (We did not see this and that means that (you just
didn’t look!)
•Special pleading (typically referring to god's will).
The baloney detection kit
•Begging the question (assuming an answer).”The stock market
fell yesterday because of a technical adjustment and profit-taking by
investors)
•Observational selection (counting the hits and forgetting the
misses).
•Statistics of small numbers (such as drawing conclusions
from inadequate sample sizes).
•Misunderstanding the nature of statistics (President Eisenhower
expressing astonishment and alarm on discovering that fully half of all
Americans have below average intelligence!)
•Inconsistency (e.g. military expenditures based on worst case
scenarios but scientific projections on environmental dangers
thriftily ignored because they are not "proved").
The baloney detection kit
•Non sequitur - "it does not follow" - the logic falls down.
•Post hoc, ergo propter hoc - "it happened after so it was
caused by" - confusion of cause and effect. The street was wet
so it has rained
•Meaningless question ("what happens when an irresistible
force meets an immovable object?).
•Excluded middle - considering only the two extremes in a
range of possibilities (making the "other side" look worse than
it really is).
•Short-term v. long-term - a subset of excluded middle ("why
pursue fundamental science when we have so huge a budget
deficit?").
3
The baloney detection kit
•Slippery slope - a subset of excluded middle - unwarranted
extrapolation of the effects (give an inch and they will take a
mile).
•Confusion of correlation and causation.
•Straw man - caricaturing (or stereotyping) a position to make
it easier to attack..
•Suppressed evidence or half-truths.
•Weasel words - for example, use of euphemisms for war such
as "police action" to get around limitations on Presidential
powers. “ or “salary adjustment”
you must ask the following 10 questions
1. How reliable is the source of the claim?
These 10 Questions and subtexts are from Michael Shermer’s
article “Baloney Detection in Scientific American 16 November
2001 , own words are in [ ]
• a given newspaper or TV channel
• a published book (not peer reviewed)
• a published article (not peer reviewed)
• a published article in a scientific journal (peer reviewed)
• advertisement
Pseudoscientists often appear quite reliable, but when examined closely, the facts
and figures they cite are distorted, taken out of context or occasionally even
fabricated. [ A couple of published books (look at the self-improvement shelves),
contain claims that are not backed up by cited work]
you must ask the following questions
2. Does this source often make similar claims?
“Pseudoscientists have a habit of going well beyond the facts.
Flood geologists (creationists who believe that Noah's flood
can account for many of the earth's geologic formations)
consistently make outrageous claims that bear no relation to
geological science.
Of course, some great thinkers do frequently go beyond the
data in their creative speculations. Their peers might think they
are wrong but not that they are cranks.”
4
you must ask the following questions
3. Have the claims been verified by another source?
“Typically pseudoscientists make statements that are
unverified or verified only by a source within their own belief
circle. We must ask, Who is checking the claims, and even
who is checking the checkers?
Example cold fusion: Stanley Pons and Martin Fleischman announced their
spectacular discovery at a press conference before other laboratories
verified it. Worse, when cold fusion was not replicated, they continued to
cling to their claim. Outside verification is crucial to good science.”
you must ask the following questions
4. How does the claim fit with what we know about how
the world works?
“An extraordinary claim must be placed into a larger context to
see how it fits. When people claim that the Egyptian pyramids
and the Sphinx were built more than 10,000 years ago by an
unknown, advanced race, they are not presenting any context
for that earlier civilization.
Where are the rest of the artifacts of those people? Where are their works
of art, their weapons, their clothing, their tools, their trash? Archaeology
simply does not operate this way. “
you must ask the following questions
5. Has anyone gone out of the way to disprove the claim,
or has only supportive evidence been sought?
“This is the confirmation bias, or the tendency to seek
confirmatory evidence and to reject or ignore disconfirmatory
evidence. The confirmation bias is powerful, pervasive and
almost impossible for any of us to avoid. It is why the methods
of science that emphasize checking and rechecking,
verification and replication, and especially attempts to falsify a
claim, are so critical. “
5
you must ask the following questions
6. Does the preponderance of evidence point to the
claimant's conclusion or to a different one?
“The theory of evolution, for example, is proved through a
convergence of evidence from a number of independent lines
of inquiry. No one fossil, no one piece of biological or
paleontological evidence has "evolution" written on it; instead
tens of thousands of evidentiary bits add up to a story of the
evolution of life.
Creationists conveniently ignore this confluence, focusing
instead on trivial anomalies or currently unexplained
phenomena in the history of life.”
you must ask the following questions
7. Is the claimant employing the accepted rules of reason and tools of
research, or have these been abandoned in favor of others that lead
to the desired conclusion?
“A clear distinction can be made between SETI (Search for Extraterrestrial
Intelligence) scientists and UFOlogists. SETI scientists begin with the null
hypothesis that ETIs do not exist and that they must provide concrete
evidence before making the extraordinary claim that we are not alone in the
universe.
In contrast, UFOlogists begin with the positive hypothesis that ETIs exist
and have visited us, then employ questionable research techniques to
support that belief, such as hypnotic regression (revelations of abduction
experiences), anecdotal reasoning (countless stories of UFO sightings),
conspiratorial thinking (governmental cover-ups of alien encounters), lowquality visual evidence (blurry photographs and grainy videos), and
anomalistic thinking (atmospheric anomalies and visual misperceptions by
eyewitnesses)”.
you must ask the following questions
8. Is the claimant providing an [own (scientific]
explanation for the observed phenomena or merely
denying the existing explanation?
“This is a classic debate strategy- criticize your opponent and
never affirm what you believe yourself to avoid criticism. It is
next to impossible to get creationists to offer an explanation for
life (other than "God did it").
Intelligent Design (ID) creationists have done no better,
picking away at weaknesses in scientific explanations for
difficult problems and offering in their stead "ID did it." This
stratagem is unacceptable in science.”
6
you must ask the following questions
9. If the claimant proffers a new explanation, does it
account for as many phenomena as the old explanation
did?
“Many HIV/AIDS skeptics argue that lifestyle causes AIDS [like
only gay men get it, wrong life style]. Yet their alternative
theory does not explain nearly as much of the data as the HIV
theory does.
To make their argument, they must ignore the diverse
evidence in support of HIV as the causal vector in AIDS while
ignoring the significant correlation between the rise in AIDS
among hemophiliacs [most of them who were heterosexual]
shortly after HIV was inadvertently [also deliberately by
companies using cheaper sources of blood] introduced into
the blood supply.”
you must ask the following questions
10. Do the claimant's personal beliefs and biases drive the
conclusions, or vice versa?
“All scientists hold social, political and ideological beliefs that
could potentially slant their interpretations of the data, but how
do those biases and beliefs affect their research in practice?
Usually during the peer-review system, such biases and
beliefs are rooted out, or the paper or book is rejected. “
These 10 Questions and subtexts are from Michael Shermer’s
article “Baloney Detection in Scientific American 16 November
2001 , own words are in [ ]
The 11th question (M. Raghunath)
11. Qui bono ? (who does benefit from other
people believing the claim ?)
Are there commercial interests behind this claims
leading to a line of research or claimed results
Does this foster the advancement of a field,
contribute to knowledge or does this just lead to the
filling on someone’s purse ?
For this very reason, reputable journals demand from authors to disclose any
conflicts of interest.
Classical example: scientists that publish about a pharmacological substance and
are shareholders of this company making this drug OR clinicians conducting trials
while being consultants or board member of the company making this drug.
7
Example black chicken
Do you know what is
said about it ?
“Black chicken is
particularly nutricious”
• what is nutricious ? How is it defined
• how are the alleged effects proven ?
• can a single food stuff help in so many
different disorders ? (anti-aging, antidiabetes, anti-autism)
8
The currently ( 14 August 2008) 16 publications on silky fowl in PubMed focus
merely on genetic and histological abnormalities. However, Toyosaki’s work (2004
and 2007) might point out to a highter degree of unsaturated fat in silky fowl eggs
with antioxidant properties.
Other work that has appeared as an abstract has not been published in full yet.
And 100 SGD go to the first NUS student
That shows me convincing – peer reviewed scientific proof published on this often used
claim (next slide) on how much knowledge is
retained hearing it, seing it, doing it etc.
bierm@nus.edu.sg
This claim is been used over and over again in seminars all
over the world, but on which data is this claim based on ?
Retention of Information according to its Presentation
90%
do it yourself
70%
tell yourself
audio-visual
50%
30%
visual
audio
Is this really so, where is
the data ?
20%
read 10%
0
20
40
60
80
100
9
The Meaning of your research training
To fit you out with a
baloney detection kit
10