Quantifying Uncertainty in Forensic
Identification
Sargur N. Srihari
University at Buffalo, The State University of New York
USA
IWBF, Valletta, Malta
March 27, 2014
1
Plan of Discussion
• Forensic Science
• Factors Leading to NAS Report
• Characterizing Uncertainty
– Similarity and Rarity
– Likelihood Ratio
• Concluding Remarks
2
Forensics and Biometrics
• Forensic Sciences:
– Concerned with crimes committed
– Identifying people/objects and
reconstructing events
– Human expert is pre-eminent
• Biometric Techniques:
– Concerned with security, preventing crime
– Identifying people only
– Automation
3
World of Crime is Complex
• Take place in
– Work, school, home, business, auto, street, internet
– Day/night, Rural/urban,Transcend borders
• Weapons are used
– Handgun, knife, blunt object
– Perpetrator
• Influence of alcohol/ illicit drugs
• Property
– Stolen, damaged
• Information Technology
– Identity theft, cyber crime
4
Crime Scene is Rich in Information
• Reveals
– Nature of criminal activity
– Identities of people involved
• Perpetrators and victims leave behind:
– Blood, saliva, skin cells, hair, fingerprints,
footprints, tire prints, clothing, fibers, digital and
photographic images, audio data, handwriting,
linguistics
• Residual effects of
– Arson, gunshots, unlawful entry
5
Evidence Collected by CSIs
• Impression Evidence:
materials with characteristics of
impressed objects
– Footwear impressions
• Biological Evidence
– DNA
– Blood type
– latent fingerprints
– Trace Evidence
– Glass
– Handwriting samples
– Fiber
– Hair
– Pollen
6
Goal of Forensic Analysis
1. Start with evidence
2. Uncover actions or happenings by:
– Identification (categorization)
• Individualization
– Association
• Between source and target
– Reconstruction
• Series of events
3. Also used for excluding
individuals/sources
7
Forensic Science Disciplines
Toxicology
Firearms/tool marks
Questioned documents
Trace evidence
Controlled substances
Biological/serology(incl. DNA)
Fire debris/arson analysis
Impression evidence
Blood pattern analysis
Crime scene investigation
Medico-legal death investigation
Digital evidence
Wide variability of:
Techniques
Methodologies
Reliability
Level of error
Research
General acceptability
Published material
8
Identification in Forensics
LaboratoryBased
People
Objects
Expert
interpretation of
patterns
Latent prints,
QD, voice,
photo, video
Trace:
paints, glass,
pollen
Impressions:
footwear, tire
treads
• Commonality with Biometrics
– Identifying people
9
Individualization
• Typical modalities
– Shoe and tire impressions, dermal ridge prints,
toolmarks, firearms and handwriting
• Assumptions
– Unique markings acquired by source randomly
– Uniqueness faithfully transmitted: source to
evidence
– Evidence originated from that source to the
exclusion of all other possible sources
10
Daubert vs Merrell Dow (1993)
• Whether Scientifically Valid
– Four factors for trial judges to consider
– One of these came from spectrographic
voice identification
• Rate of false positives and false negatives
– Court should consider error rates
• Rules of evidence in many states
– General acceptance rather than scientific
validity
11
The Brandon Mayfield Case
1.
2004 Madrid train bombings
–
–
–
1.
Spanish National Police (SNP) recovered a latent print (LFP 17)
–
2.
Though Mayfield had never been to Spain, Passport had expired
Mayfield
(15 level 2
similarities)
Algerian with criminal record, Spanish residency, Terrorist links
FBI concluded that earlier individualization was in error
–
6.
Prints from 1984 when arrested for burglary as a teenager
Confirmed by 3 FBI examiners + outside consultant
Converted to Islam, married an Egyptian
Represented man in child custody who turned out to be a jihadist
Meanwhile, SNP sourced LFP 17 to Ounahne Daoud
–
5.
Partial on a plastic bag of detonators in van used by perpetrators
On May 6, 2004, FBI arrested Mayfield
–
4.
LFP 17
(Evidence)
FBI IAFIS: LFP 17 has match with Mayfield, Oregon lawyer
–
–
–
–
3.
On the morning of March 11, 2004
Series of bombings against commuter train system (4 trains)
Killing 191 people and injuring 2,000 others
Released in 3 weeks, $2 million for mistake with apology
Independent Review
–
–
Not quality of images
Bias and Circular reasoning
Daoud
12
National Academy of Sciences
Committee on identifying the needs of the forensic science community (2007-09)
1. HARRY T. EDWARDS, (Co-chair), Judge, U.S. Court of Appeals, District of Columbia Circuit
2. CONSTANTINE GATSONIS, (Co-chair), Director, Center for Statistical Sciences, Brown University
3.
4.
5.
6.
7.
MARGARET A. BERGER, Suzanne J. and Norman Miles Professor of Law, Brooklyn Law School
JOE S. CECIL, Project Director, Program on Scientific and Technical Evidence, Federal Judicial Center
M. BONNER DENTON, Professor of Chemistry, University of Arizona
MARCELLA F. FIERRO, Medical Examiner of Virginia
KAREN KAFADAR, Rudy Professor of Statistics and Physics, Indiana University
8. PETE M. MARONE, Director, Virginia Department of Forensic Science
9. GEOFFREY S. MEARNS, Dean, Cleveland-Marshall College of Law, Cleveland State University
10. RANDALL S. MURCH, Director, Research, Virginia Polytechnic Institute and State University
11. CHANNING ROBERTSON, Bowes Professor, Dean of Faculty, Dept Chemical Engg, Stanford University
12. MARVIN E. SCHECHTER, Attorney
13. ROBERT SHALER, Professor, Biochemistry, The Pennsylvania State University
14. JAY A. SIEGEL, Professor, Forensic Program, Indiana University-Purdue University
15. SARGUR N. SRIHARI, SUNY Distinguished Professor, Dept Computer Scien & Engg, University at Buffalo
16. SHELDON M. WIEDERHORN (NAE), Senior Fellow, National Institute of Standards and Technology
17. ROSS E. ZUMWALT, Chief Medical Examiner, State of New Mexico
13
NAS Committee Report
Released March 2009
National Academies Press
Committee at
NAS, Woods Hole, MA
14
Determination of Uniqueness
• Requires
1.Measurement of object attributes
– Determine distribution of attributes
– Testing of attribute independence
2.Calculation of probability that different objects
share attributes
– Similarity
15
Forensic Expert Opinion
• Three possible opinions for evidence
• Individualization
– No other individual on earth is source of mark
• Inconclusive
• Exclusion
– Definitely not this individual
• No method for characterizing uncertainty
16
Methodology Needed
1. Model expert intuition on rarity
– Locard: Finger print features should be
assessed relative to their rarity
2. Similarity is commonly used in
Biometrics
• How to reconcile both
17
Rarity Metrics: PRC in database of size n
PRC
nPRC
Conditional nPRC
For identical match
Rare
Common
nPRC=1.17 x10-5
nPRC=0.156
18
nPRC=2.14 x 10-8
nPRC=0.166
Likelihood Ratio
• h0: Evidence (with characteristics) e was
generated by known (individual with
characteristics) k
• h1: Evidence e was not generated by k
0
p(k,e | h )
Likelihood Ratio: LR(k,e) =
1
p(k,e | h )
Probability of evidence
and known (suspect) under
Prosecution hypothesis
Probability of evidence
and known (suspect) under
19
Defense hypothesis
From LR to Probability
• Can determine probability of identification
p(h | k,e) =
0
O posterior
1+ O posterior
• Under equal priors, or prior odds even
p(h 0 | k,e) = sigmoid(LLR(k,e))
– Mapping from LLR to probability
– Can fuse results from different modalities
20
Prior/Posterior Odds and Population
• If LR(k,e) = 106
• Same source is million times more probable than different
Population Size, n
Posterior Odds
With LR= 1,000,000
World (7,000,000,000)
1:7,000
USA (300,000,000)
1:300
0.0033
NYC (8,000,000)
1:8
0.1111
Colorado Springs
(400,000)
2.5 : 1
0.7143
Walla Walla (30,000)
33:1
College Dormitory (200)
5,000:1
P(h0)
0.0001
Individualizaton
implies
P(h0)=1
0.9706
21
0.9998
Computing the Likelihood Ratio
• Gaussian case offers some insights
• Shattered Glass
– Refractive index is a continuous scalar
22
LR with Gaussian distributions
Each glass type
is imperfect--has
a Gaussian distribution
Different glass types
manufactured with
a Gaussian distribution
Glass refractive index μ
p(e | θ ) ~ N( μ1,σ 2 )
h 0 : μ1 = μ2
p(k | θ ) ~ N( μ2 ,σ 2 )
p(θ ) ~ N( μ, τ 2 )
h1 : μ1 ≠ μ2
Probability of Prosection hypothesis
p(k,e | h 0 ) = ∫ p(k | θ )p(e | θ ) p(θ )dθ ,
since k and e are two samples randomly picked from the same distribution
Probability of Defense hypothesis
p(k,e | h1 ) = ∫ p(k | θ1 )p(θ1 )dθ1 ∫ p(e | θ 2 )p(θ 2 )dθ 2
since k and e are independently picked from two distributions
23
LR Factorization (Gaussian)
∫ p(k | θ )p(e | θ )p(θ )dθ
p(k, e | h ) = ∫ p(k | θ )p(θ )dθ ∫ p(e | θ )p(θ )dθ
p(k, e | h 0 ) =
1
1
p(e) ~ N( μ1 ,σ 2 ),
p(k) ~ N( μ2 ,σ 2 )
1
1
2
2
2
p(θ ) ~ N( μ, τ 2 )
τ
2
2
⎧
⎫
⎧
⎫
(k
−
e)
μ
)
(w
−
k+e
2
2
LR(k,e | σ , μ, τ ) =
exp ⎨−
exp ⎨
⎬ where w =
2 ⎬
2
2
2σ
⎩ 4σ ⎭
⎩ 2τ
⎭
Increases with Similarity (k-e) and Rarity
Rarity is inverse probability (w − μ )
24
Similarity and Rarity
• Also commonly used in Information Retrieval
• To score a user query q represented by term (t)
against a document (d) in a database (D)
– TF (term frequency)
• Measure of similarity of q and d
– IDF (inverse document frequency)
• Measure of rarity of t in D
25
Analogy to Search Engine Measure
TF-IDF
26
TF-IDF and LR Factorization
• Term Frequency-Inverse Document Frequency
– tf(t,d) × idf (t,D)
– tf= no of times term t occurs in d
– idf=divide no of docs |D| by no of docs containing term t
• Numerical value reflects importance of term to
document considering the corpus (collection)
• Equivalent to product of similarity and rarity
⎧ (k − e)2 ⎫
⎧ (m − μ )2 ⎫
k+e
LR =
exp ⎨−
exp
where
m
=
⎬
⎨
⎬
2
2
2
2σ
⎩ 4σ ⎭
⎩ 2τ
⎭
τ
27
Computational Forensic Examination
• LR factorization matches intuition
• A step towards computational forensics
– Systematizing human procedures using
computational means
• Forensic Examiners use
– Class Characteristics
– Individualizing Characteristics
• Individualizing characteristics bear more weight
– But there are computational compexity issues
28
LR and Curse of Dimensionality
0
p(k, e | h )
Likelihood Ratio: LRJ (o, e) =
p(k, e | h1 )
1.If k and e have d features each
– If each feature has K discrete values,
d=6, , K=4 or 5,
• no. of parameters needed is 2K 2d
No of parameters= 4,799
– Exponential in no. of features
2.Need samples from k & e for each hypothesis
– Impractical even with feature independence
29
Simple Solution: Distance Method
0
p(d(k, e) | h )
Likelihood Ratio: LRD (o, e) =
1
p(d(k, e) | h )
• Maps two multivariate
distributions of 2d variables
each into two univariate ones
• Severe loss of information
• Even if we use vector distance
p(d(k, e) | h0)
LRVD (k, e) =
p(d(k, e) | h1)
– Still there great information loss
30
Generalization of LR Factorization
• Generalize to mutivariate
1
LRDR = P(d(k, e) | h )*
P(m(k, e))
–d(k,e) is distance and m(k,e) is mean
0
• Distance measures
1.Binary
• Difference is 0,1 or -1
• Mean is 0 if bits are different, 1 otherwise
2.Multinomial
• Difference of categorical values
3.Graph
• Difference of features of matching nodes/edges
• Mean of feature vector of matching nodes/edges
31
We now have three LR methods
1. Based on Joint Distributions
p(k, e | h 0 )
LRJ (o, e) =
p(k, e | h1 )
2. Based on Distance Distributions
0
p(d(k, e) | h )
LRD (k, e) =
p(d(k, e) | h1 )
3. Based on Distance and Rarity Distributions
1
LRDR (k, e) = P(d(k, e) | h0)*
P(m(k, e))
32
Comparison of Error Rates
• Determine error based on whether LLR
is positive or negative
• Gaussian distributions
Univariate
5-variate
33
Multinomial Data: Handwriting (“th”)
Joint Distribution: assumed independence due to intractability.
Three distance methods:
L: Lin
O: overall frequency
G: Goodall
34
Graph Matching: Footwear
Using
EMD
between
graphs
35
Three LR methods with six data types
LRJ, LRD, LRDR
1. Uni Gauss
2. Mult Gauss
3. Bin Ind
4. Bin Dep
5. Multinomial
6. Graph
36
Computation of LRDR
• Need two distributions (difference and mean)
each with d variables
– No. of parameters is 2Kd (where K=no. of values)
– As opposed to 2K2d with LRJ
– We are dealing with two d-dimensional distributions
rather than two 2d dimensional distributions
• Still exponential with d: scalability is an issue
• Solution is to use PGMs of feature variables
– Bayesian networks
– Markov Networks
37
Markov Structure Learning
Manual
Manual
Modified
Chow-Liu
Greedy
L1-Reg.
Fast
Greedy.
38
Summary
1. Forensic Identification facing Controversy
– Expressing Uncertainty is necessary
2. LR using Joint distributions: exact, intractable
– Distance based LR are a rough approximation
3. LR factorization based on similarity-rarity
– Analogous to TF-IDF
– Matches human expert intuition
•
Computational thinking
– more accurate but still intractable
•
PGMs provide a solution
– Markov network structure learning
39
Relevant Paper
40