Stories and statistics:
What can the Sally Clark case
tell us about the psychology of
evidential reasoning?
David Lagnado
Division of Psychology and Language
Sciences
University College London
Evidential reasoning
 How do people assess and combine evidence to
make decisions?
– Legal, Medical, Financial, Social …
 Cognitive science approach
– What kinds of representations?
– What kinds of inference patterns?
 How do these compare with normative or formal
methods of evidential reasoning?
– Bayesian networks now used to model complex
forensic evidence (Taroni et al, 2006)
Reasoning with legal evidence
Legal domain
– E.g. juror, judge, investigator, media
Complex bodies of interrelated evidence
– Forensic evidence; witness testimony; alibis;
confessions etc
Need to integrate wide variety of evidence
to reach singular conclusion (e.g. guilt of
suspect)
Story model
(Pennington & Hastie, 1986, 1991, 1992)
 Evidence evaluated through story construction
– ‘Stories involve human action sequences in which
relationships of physical causality and intentional
causality between events are central’
 Jurors use prior causal knowledge and
expectations about story structure to fill in gaps
in evidence
 Active ‘sense-making’ process to construct an
account of what happened
Evidential Reasoning
 Reasoning from
evidence
– Use the evidence to
construct ‘most
plausible’ account of
what happened
– Generate a causal
story based on the
evidence
 Reasoning about
evidence
– Assessing the
strength/reliability/vali
dity of the evidence
– How well does the
evidence support the
putative
hypotheses/stories?
think-aloud protocols from jurors in
simulated trials suggest that they
predominantly engage in former
Continuum?
 Individual variability in competence at juror reasoning
(Kuhn et al., 1994) and evidential reasoning in general
(Kuhn, 1991)
SATISFICING
Construct
single story
using evidence
THEORY-EVIDENCE
CO-ORDINATION
Construct multiple stories
Evaluate against evidence
and alternatives
Group deliberation helps shift --->
Requires ability to
reflect on one’s own
reasoning?
Stories: blessing or curse?
 Story is concrete and categorical
 Describes a singular causal process
 Economy of representation
 Easy to communicate
 Clear-cut basis for decision and
action
 Identify key variables to blame
 Hard to simultaneously
compare/evaluate multiple
stories (cf Wigmore)
 Danger of neglecting alternative
accounts
 Evidence often
gathered/interpreted for a single
story (confirmation bias)
 The ‘truth’ might not make a
good story
Binocular rivalry
Switch
between two
coherent
percepts
(green vs red)
Even when
inputs are
mixed
Switch
between two
coherent
stories
(prosecution
vs defence)
Even when
evidence is
mixed
Sally Clark case
 Sally & Stephen Clark married, both solicitors
 Son Christopher born in 1996
– Died suddenly at home aged 11 weeks
– Sally alone with child; noticed he was unwell;
ambulance called, but he could not be resuscitated
 Postmortem (Dr Williams):
– Death from natural causes - lung infection (and
bruises consistent with resuscitation attempts)
– Body was cremated
Sally Clark case
 Harry born in 1997
– Died suddenly at home aged 8 weeks
– Stephen at home with Sally; but Sally alone with child when
discovered unwell; ambulance called, but he could not be
resuscitated
 Postmortem (Dr Williams):
– Suspicious - death from shaking?
– Re-examined death of Christopher
– Concluded it too was unnatural, with evidence of smothering
 Sally Clark charged with murder of both children
Prosecution case
 Christopher & Harry were smothered
– Nb change from Dr Williams’ initial claims of shaking
for Harry (error in diagnosis of retinal haemorrhages)
 Neither died from SIDS because there were
unexplained injuries
 Numerous similarities between the two deaths
– ‘which would make it an affront to commonsense to
conclude that either death was natural, and it was
beyond coincidence for history to so repeat itself’
Prosecution case
 ‘Similarities’
– Babies died at similar ages
– Both found unconscious in same room; at same time;
shortly after feed
– Mother alone with child when found unwell
– Father either away or due to go away
– (Medical evidence of previous abuse & deliberate
injury)
 How unlikely are these given that mother is
innocent? (beyond coincidence?)
Prosecution case: Injuries to
Christopher
Prior
smothering
Smothering
nosebleed
Blood in
lungs
Torn
frenulum
Both fresh &
older blood
Between lip
and jaw
Bruises
Small marks
on arms and
legs
Prosecution case:
Injuries to Harry
Shaking/
prior abuse
Smothering
Hypoxic
damage to
brain
Haemorrhage
s in eyelids
Haemorrhage
s to eyes
Rib injuries
Old fracture &
dislocation
Spinal
injuries
Spinal
bleeding &
swollen cord
Prosecution case:
Credibility of witnesses
Sally’s testimony in doubt
Sally Clark
reliability
Sally Clark states
she found Harry
slumped in bouncy
chair
Harry slumped
in bouncy
chair?
Police surgeon says
impossible for baby
of 8 weeks to slump
in bouncy chair
Stephen’s testimony in doubt
Stephen lying
to protect
wife
Stephen
Clark
reliability
Stephen Clark states
he returned home at
5.30/5.45pm
Sally Clark
smothered
Harry
Opportunity
/ Motive
Sally Clark
alone with
Harry
Taxi records show
Stephen Clark
returned home at
8.10pm
Prosecution case:
Statistical evidence
 Professor Sir Roy Meadow (Paediatrics)
 Report – ‘Sudden unexpected deaths in
infancy’
 Risk factors – age of mother (<26), smoker in
household, no wage earner
 None applied to Clark family
 Chance of one SIDS in family= 1 in 8,543
 Chance of two SIDS = 1/8543 x 1/8543
= 1/73 million
 ‘…by chance that happening will occur
about once every hundred years’
Defence case
 Sally Clark did not kill her children
– They died of natural but unexplained causes
– Medical evidence amounts only to suspicion
 Two of prosecution experts said cause of deaths
‘unascertained’
 Case hinges upon Dr William’s reliability and
competence
Defence case: Injuries to Christopher
Haemoderosis
Blood in
lungs
Change of opinion
Poor conduct of postmortem
Low quality photos etc
Resuscitation
attempts
Postmortem
effects
Torn
frenulum
Bruises
Report of
Torn
frenulum
Report of
Bruises
Reliability of
Dr Williams
NB distinguish
event from
reports of
event
Report from
police &
hospital
Defence case: Injuries to Harry
Natural
causes
post-death
Hypoxic
damage to
brain
Postmortem
Haemorrhage
s to eyelids
Haemorrhage
s to eyes
Change of opinion
Prior error with slides
Rib injuries
Reliability of
Dr Williams
Spinal
injuries
Explain Stephen testimony mistake
Stephen very
unlikely to lie to
protect wife if
she killed their
children
Stephen
Clark
reliability
Stephen
admitted lack
of knowledge,
and
mentioned taxi
records
Sally Clark
smothered
Harry
Opportunity
/ Motive
Sally Clark
alone with
Harry
Stephen Clark states
he returned home at
5.30/5.45pm
Taxi records show
Stephen Clark
returned home at
8.10pm
Defence case: Statistical evidence
Mother >26
No smokers
Wage earner
Known risk
factors
Estimate for
probability of
one SIDS death SIDS death1
questionable
UNKNOWN risk
factors
SIDS death2
genetic or
environmental
factors
Calculation for
two deaths
ignores possible
genetic &
environmental
factors
Deaths are not
independent (so cannot
simply square)
2 SIDS death = significantly greater than 1/73 million
Verdict
 Sally Clark found guilty by 10-2 majority
 Imprisoned for life
First Appeal:
Statistical evidence misleading
 Non independence
– 1/73 million figure flawed
– Probabilities are not independent
 Relevance
– Probability of two SIDS deaths insufficient
– needs to be compared against probability that mother murders
both her children
– Estimated incidence of this is much lower than of two SIDS
deaths
‘it is clearly inadequate to concentrate on a single cause of death. If we
make an assessment of the probability of two babies in one family both
dying from SIDS, we must equally make a similar assessment of the
probability of two babies in one family both being murdered (and so on, for
any other causes that may be under consideration)…’ Dawid (2002)
 Two alternative causes of the deaths
(exclusive but not exhaustive – other causes possible, also
possible that one SIDS, one murdered etc)
Prior
probability
of SIDS is
low
SIDS
Murder
Evidence
Prior
probability
of murder
is even
lower
Evidence of 2 deaths
Prior to other/medical evidence, probability of double SIDS greater
than probability of double murder
Appeal dismissed
 Court of appeal judgment
– No need for expert statisticians to give oral testimony –“it was
hardly rocket science”
– Defence already pointed out flaws in statistics
– What matters is that probability of two SIDS deaths is very low,
not exact figure
– Statistic might have had larger impact on jury than it should
have, but case against Sally Clark was nevertheless
overwhelming
 "In the context of the trial as a whole, the point on statistics was of
minimal significance and there is no possibility of the jury having
been misled so as to reach verdicts that they might not otherwise
have reached."
Second appeal
 Discovery of new evidence
– Harry had bacterial infection
– Known by Dr Williams but not disclosed at trial!
– (When jury asked about blood tests for Harry,
Williams said no relevant test results)
 Plausible cause of Harry’s death
– according to 11 independent experts
– Also casts doubt on Christopher’s death due to
unreliability of Dr Williams
Harry’s death
Natural
causes
post-death
Hypoxic
damage to
brain
Bacterial
infection
Microbiological
tests
Hemorrhages
to eyelids
Failure to
disclose
etc
Postmortem
Hemorrhages
to eyes
Rib injuries
Spinal injuries
Reliability of
Dr Williams
Conclusions
about Christopher
Second appeal
 Conviction declared unsafe
– Sally Clark released 2003
 Postscript
– Several other similar convictions involving Meadow
subsequently overturned
– Meadow struck off medical register 2005; reinstated
on appeal 2006
– Williams guilty of serious misconduct 2005
 Sally dies 2007
Lessons
 Various repercussions for legal domain
– Expert witnesses
– (expert in child health not an expert in statistics)
– Interpretation and presentation of statistical evidence
 For evidential reasoning
–
–
–
–
Understanding statistical evidence
Role of causal networks
Reliability of evidence (and experts)
Stories and blame
Statistical evidence
 Well-documented problems when people reason with
probabilities (Kahneman, 2012)
– Base rate neglect; prosecutor's fallacy; conjunction errors
 In contrast people are good at qualitative causal
reasoning
 One approach that reconciles these findings
– People need suitable causal models for appropriate probabilistic
reasoning (Krynski & Tenenbaum, 2007; Sloman, 2005;
Lagnado, 2011)
 Classic probability problems facilitated with causal
models
Medical diagnosis problem
(Krynski & Tenenbaum, 2007)
Cancer
+ TEST
 Given +test people grossly
overestimate probability of
cancer
 (Neglect low base rate)
 Mistaken use of false positive
probability
 Low false +  high probability
of cancer
Cyst
Cancer
+ TEST
 Alternative cause of +test
made explicit
 People give better estimates of
probability of cancer
 Improved probabilistic
reasoning given suitable causal
model
 shown for several classic
problems
Statistical evidence
 To avoid errors in Sally Clark case
– Need suitable (causal) model to
understand probabilities
– Need to consider (probability) of
alternative causes
– Need to combine via Bayes rule
Misleading categories
Case framed as murder vs SIDS
Exclusive but not exhaustive
Tempting to reason: not-SIDS -> murder
But other natural explanations possible (eg infections
etc)
 Key to represent alternative causes …




smother
other natural
Natural
Unnatural
Evidence
Other …
SIDS
Evidence
Non-independence
 Main focus on flawed assumption of independence of
SIDS deaths
– Judges, lawyers, media, etc
 People understand independence/non-independence
when framed causally
– Possible unobserved common causes of SIDS deaths
– Eg genetic or environmental factors
Genetic or
environmental
SIDS1
SIDS2
Understanding/using
probability
 Second error –
– How is probability of SIDS relevant to probability that
Sally is guilty of murder?
– Need to use Bayes rule
– Requires comparison with prior probability of child
murder
 Danger of prosecutor's fallacy
– Assume that 1 in 73 million figure applies to
probability that Sally Clark is innocent
– Eg P(2deaths|not guilty) = P(not guilty|2deaths)
Statistical evidence
 Probabilistic reasoning improved by explicit causal
models (Krynski &Tenenbaum,2007)
 Avoid Meadow’s second error by explicitly representing
probability of double murder?
Prior of
double
SIDS is
low
SIDS
Murder
Evidence
Prior for
double
murder is
even lower
Representing alternative cause and its prior
probability should improve probabilistic judgments
Ongoing empirical work on improving Bayesian
reasoning using causal models
Causal networks
 Key role of causal reasoning borne out by Sally
Clark case
 But story model needs to be developed
 Formal means for representing causal models
and inference
 Include representation of evidence and reliability
(and their interrelations)
 Move closer to theory-evidence co-ordination
 Even if people don’t always do this- they can!
Legal idioms
 Evidential reasoning in terms of causal
building blocks
– Capture generic inference patterns
– Reusable and combinable
– Qualitative causal structure
– Based on Bayesian networks
– Akin to schema/scripts
Fenton, Lagnado & Neil, 2012
Legal idioms
 Evidence idiom
 Evidence depends on Hypothesis
 Evidence is more likely if hypothesis is true
 Observed evidence raises the probability of hypothesis
Hypothesis
Smother
Evidence
Bruises
Smothering
causes bruises
(probabilistically)
Legal idioms
 Explaining away
– Evidence is often rebutted
Smother
Resuscitation
Bruises
Stephen report
Evidence for
alternative
cause of bruises
Legal idioms
 Distinguish event from report
Hypothesis
Christopher
smothered
Event
Bruises
Report
Williams report
of bruises
Police / hospital
report of NO
bruises
Legal idioms
 Evidence – Reliability idiom
Hypothesis H
Reliability
Impact of evidence on
hypothesis is
modulated by its
reliability
Evidence E
Bruises
Williams report
Reliability of
Williams
Williams slide
errors
Legal idioms
 Reliability of witness reports
– Separate factors for reliability
Is Williams mistaken?
Competence
Is Williams biased?
Objectivity
Is Williams honest?
Veracity
Reliability of
Williams
Bruises
Williams report
From Schum (2001)
Legal idioms
 Opportunity idiom
Sally alone
with Harry
Sally smothers
Harry
Reliability
Stephen report
Opportunity is often a pre-condition of guilt
Legal idioms
 Motive idiom
Sally career
driven
Sally resentful
Use of irony
rebuttal
Sally murders
baby
Letters to
parents
evidence
Motive is typically a
pre-condition of guilt
Combining idioms – alibi evidence
Opportunity
Stephen lying
to protect
wife?
Sally alone
with Harry
Sally smothered
Harry
Conflicting
Evidence
reports
Reliability
Stephen
memory
error?
Stephen report
5.30
Taxi record 8.10
Status of framework
 Normative
– Formal model to capture appropriate probabilistic
inference (and support theory-evidence co-ordination)
 Descriptive
– Do people’s inferences conform to the model?
– Qualitatively? Quantitatively?
– Empirical studies suggest good fit to qualitative
patterns
 Prescriptive
– Guide to interpreting complex evidence and improving
inference (shift towards TEC)
The big picture
 Combining network fragments into a
large-scale model
Key factors at trial
Prosecution case
Defence case
Cognitive Economy?
 How do people do this?
– Lab-based studies support the claim that they use
idioms for small-scale problems (Lagnado, 2011;
Lagnado et al., 2012)
– But how does this scale-up?
 Story-telling
– Use of narrative to simplify?
– Reasoning from but not about evidence
Stories and Blame




Stories constructed from causal networks
Cohesive narrative to explain events
To attribute blame for negative outcomes
But focus of stories can compromise proper
theory-evidence co-ordination
Stories and Blame
 At trial
 Prosecution presented one cohesive story
– Sally smothered both babies
 Explains most of the medical evidence
 Explains unreliability of Stephen & Sally testimony
 ‘Supported’ by statistical evidence
 Defence did not present one single story,
but numerous disconnected pieces to
explain the different injuries etc
Possible line of juror reasoning?
 Jurors reject SIDS due to extreme rarity
 Neglect low base-rate of smothering
because this was never raised at trial
 Accept smothering because:
– it gives ‘simple’ explanation of injuries
– (and explains inconsistent testimonies)
– Assigns blame to someone
 A ‘plausible’ story?
Stories and Blame
 Importance of causal story that assigns
blame?
 At second appeal
– New ‘story’ in which Harry died from
infection and Dr Williams & Meadow were
blamed
 Aftermath & Media
– Professor Meadow’s statistical errors are
highlighted
Lessons for evidential
reasoning
 Importance of clarity in evidential reasoning
– For jurors, lawyers, judges, experts, media …
 How can this be improved?
– Shift from single casual story to theory-evidence coordination
– Use people’s capacity for causal reasoning to support
better probabilistic inference?
– Introduce formal methods eg Bayesian networks etc
to help model and evaluate evidence?
– Ongoing research!
Thank you!
 Collaborators
– Norman Fenton (QMUL)
– Martin Neil (QMUL)
 Evidence project
– Philip Dawid
– William Twining
BAYES RULE (odds version)
P(guilty | 2deaths)
P(2deaths | guilty) P(guilty)
=
×
P(~ guilty | 2deaths) P(2deaths |~ guilty) P(~ guilty)
P(2deaths|guilty) = 1
P(2deaths|~guilty) =1/73million (ignoring error of
non-independence)
P(guilty) = 1/84million (based on stats for double
child murders – but perhaps should just consider
guilty = at least one murder)
Nb
P(guilty|2deaths) = 0.009
p/1-p = odds
P = odds/(1+odds)