Error Rates and Random Match
Probabilities (RMP) Based on the
RUGER 10-Barrel Test and the
GLOCK Cartridge Case Tests
James E. Hamby, Ph.D., David J. Brundage,
M.S., Steven A. Norris, B.S., Nicholas D.K.
Petraco, Ph.D., and James W. Thorpe, Ph.D.
James E. Hamby, Ph.D.,
International Forensic Science Laboratory &
Training Centre, Indianapolis, Indiana
David J. Brundage, M.S.,
Forensic Science Consultant, Nashville,
Tennessee
James W. Thorpe, Ph.D.,
University of Strathclyde, Glasgow, Scotland
(Retired)
A total of 240 test sets were produced by Dave
Brundage, Mickey French and Jim Hamby – all then
employed at the Indianapolis-Marion County
Forensic Laboratory, Indianapolis, IN
Recently 20 additional test sets were produced by
Hamby and 10 sets sent to Europe and other
countries for examination. The various test sets
have used 4 different types of 9mm ammunition.
Of the 626 participants in this research project, nine
have used some type of ‘ballistics’ imaging
equipment.
Of the 626 participants in this worldwide
project,9,382 of the possible 9,390
unknown bullets were identified. (15
unknowns x 626 participants = 9,390)
Three bullets were mutilated during test
firing (tank rash) and five ‘unknown’ bullets
could not be associated to the known
bullets by two young examiners-in-training
- listed as inconclusive.
This worldwide research project and
numerous other research projects reported
in the scientific literature – the AFTE
Journal, the Journal of Forensic Sciences,
and others - since the early 1930’s should
provide data that will obviate some of the
concerns within the legal community especially in the United States.
• We request that those members who
haven’t participated ‘take’ the test in their
laboratory & submit their answers!
• We also request that you return the set if
you are no longer using it as there are labs
that have requested test sets. Some labs
may have two sets in their possession.
13
Modified SWGGUN Power Point
Using the excellent SWGGUN power
point, I have made a few changes that
work well when testifying especially in
Frye or Daubert Hearings. The
following are a few of the changes.
Our beloved AV man–Mike–has the
power point if you are interested.
Definition: Subjective Examination
The results of a subjective examination are
based on an individual’s opinion based on
his/her training and experience. This does
not mean that this type of examination is
unreliable or unscientific.
There is subjectivity in every science and in
every test, whether it is a doctor diagnosing
a head cold, a physician examining an x-ray
or a cytotechnologist examining slides for
cancer cells (pap smears).
• National Institute of Justice, Washington, D.C.
•
NIJ - in conjunction with the National Forensic Science Technology Center
(NFSTC) – funded a distance learning program titled ‘Firearms Examiner
Training Program’ to augment training for new examiners,
• Federal Bureau of Investigation (FBI)
•
Since 1986, the FBI has offered a one week course titled ‘Specialized
Techniques in Firearms Identification’. The course has been attended by
several hundred firearms examiners.
• Bureau of Alcohol, Tobacco and Firearms (ATF)
•
Since 1999, the ATF has offered a one-year long course titled the ‘National
Firearms Examiner Academy. To date, 140+ firearms examiners have
attended the NFEA. (Funded by ATF and NIJ)
• California Criminalistics Institute (CCI), CA DOJ
•
For several years, CCI has offered a variety of specialized courses in
firearms and toolmark related fields.
• United Nations Office on Drugs and Crime (UNODC)
•
•
•
•
In 2013, the UNODC – with funding from the Canadian Government – is
developing a limited service forensic laboratory for the Palestinian Civil
Police (PCP) Temporary Forensic Science Laboratory (TFSL) in Ramallah,
Palestine.. The TFSL will have the following forensic specialties:
Forensic Chemistry – Drug and Arson Analysis
Questioned Documents Analysis
Firearms and Tool Mark Identification
• United States Department of State
•
In 2014, the US Department of State – through the Bureau of International
Narcotics and Law Enforcement Affairs, US Embassy, Belize, funded a
program to strengthen the Belize National Forensic Science Laboratory by
providing technical assistance related to the handling, processing and
analysis of ballistics (firearms / tool marks) related evidence.
• Other programs such as ICITAP teach firearms and tool
mark Identification around the world.
Firearm & Toolmark Identification meets the reliability
standard put forth by both Daubert or Frye requirements.
The science has been accepted in every state and
Federal court that uses Daubert or Frye.
Examples: A recent (9-11) Indiana Supreme Court
Decision (Frye) upheld the reliability of Firearms and
Tool Mark Identification; and a recent (12-11) Kansas
District Court Decision (Frye) upheld ‘general
acceptance’ in the field. A recent (7-13) Florida Court
Issued a Daubert decision upholding the science. Cases
are available for review on the SWGGUN website (under
ARK) at www.swggun.org.
• The Good News:
o 10-Barrel Test: # examiner errors committed = 0
o GLOCK Cartridge Case Test: # of falsely
matched cartridge cases = 0
THIS SHOULD BE GREAT!
We WANT error rates of 0% for good
classification systems!!
• The Bad News:
o You have to deal with the U.S. court system
o 0% error rates present an opportunity to muddy
waters with the “unrealistic study” criticisms
o Little court interest in understanding the technicalities
of estimating small error rates.
So what can we do?
• 0% error rate is the “frequentist” estimate
o “Bayesian” statistics provide complementary
methods
o Can work much better in estimating small probabilities
o We looked to sports statistics for low scoring games
• For 10-Barrel we need to estimate a small
error rate
• For GLOCK we need to estimate a small
random match probability (RMP)
• Use Bayesian “Beta-binomial” method when
no “failures” are observed (Schuckers)
• Basic idea of the reverend Bayes:
Prior Knowledge × Data = Updated Knowledge
a
Error Rate/RMP =
a+b
U(a,b) × Beta-Binomial(data | a,b)
Posterior(a,b | data)
Get updated estimates of Error rate/RMP
• So given the observed data and assuming
“prior ignorance”
o Posterior error rate/RMP distributions:
Posterior
Distribution of RMP, 1632 Cartridge Cas
Distribution of Average Examiner Error Rate for 626
Participants
Posterior Dist. GLOCK
100
300
RMP
0.000086%
[0.0000020%, 0.00031%]
0
Frequency
100
200
300
Average Examiner Error Rate
0.011%
[0.00023%, 0.040%]
0
Frequency
400
Posterior Dist. 10-Barrel
0.00
0.02
0.04
0.06
0.08
Error Rate (%)
0.10
0e+00
2e−04
4e−04
RMP (%)
6e−04
8e−04
# of
# ofParticipants
Participants
2e−04
4e−04
GLOCK
0e+00
300 400 500 600 700 800 900
Estimated AverageRMP
Examiner
(%) Error Rate (%)
0.020
Actual:
626 participants
0.010
Error Rate
Examiner
Estimated
Examiner
Error(%)
Rate (%)
10-Barrel
Actual:
1632 cartridge cases
1000 2000 3000 4000 5000 6000
Cartridge
Cases
# Cartrige#Cases
Examined
with NoExamined
Errors in Identifica
With No Errors in I.D.