CCI Firearms and Toolmark Examiner Academy Workshop on Current Firearms and Toolmark Research 3 2 1 0 1 2 3 Pushing Out the Frontiers of Forensic Science Outline • Morning-ish • Introduction and the Daubert Standard • Confocal Microscopy • Focus Variation Microscopy • Interferometric Microscopy • Surface Data/Filtering Outline • Afternoon-ish • Similarity scores and Cross-correlation functions • Known Match/Known Non-Match Similarity Score histograms. False Positives/False Negatives/Error Rates • Multivariate Discrimination of Toolmarks • Measures of “Match Quality” • Confidence • Posterior Error Rate/Random Match Probability • Lessons learned in conducting a successful research project Introduction • DNA profiling the most successful application of statistics in forensic science. • Responsible for current interest in “raising standards” of other branches in forensics…?? • No protocols for the application of statistics to comparison of tool marks. • Our goal: application of objective, numerical computational pattern comparison to tool marks Caution: Statistics is not a panacea!!!! The Daubert Standard • Daubert (1993)- Judges are the “gatekeepers” of scientific evidence. • Must determine if the science is reliable • Has empirical testing been done? • Falsifiability • Has the science been subject to peer review? • Are there known error rates? • Is there general acceptance? • Federal Government and 26(-ish) States are “Daubert States” Tool Mark Comparison Microscope G. Petillo 4 mm G. Petillo 5/8” Consecutively manufactured chisels Known Match Comparisons G. Petillo 5/8” Consecutively manufactured chisels Known NON Match Comparisons G. Petillo 5/8” Consecutively manufactured chisels 4 mm 4 mm 600 um Confocal Microscope Marvin Minsky First confocal microscope Confocal Microscopes Source Illumination aperture Detector Confocal pinhole Out of focus light In focus light Objective lens Tool mark surface (profile of a striation pattern) Focal plane for objective Sample stage Rastering pattern of laser confocal Nipkow disk sweeps many pinholes Programmable array Illumination/Detection Get any illumination/detection pattern Objective’s focal plane Sample stage Scan stage in “z”-direction Detector Objective’s focal plane Sample stage Scan stage in “z”-direction Detector Objective’s focal plane Sample stage Scan stage in “z”-direction Detector Objective’s focal plane Sample stage Scan stage in “z”-direction Detector Objective’s focal plane Sample stage Scan stage in “z”-direction Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector Objective’s focal plane Scan stage in “z”-direction Sample stage Detector For Each Detector Pixel: Intensity Axial Response Function Point on surface corresponding to pixel’s is in maximum focus here z Record the “axial response” as stage is moved along the z-direction Increasing surface height All-in-Focus 2D Image Overlay confocal “z-stack” • 3D confocal image of portion of chisel striation pattern Confocal Microscope Trivia • Use high NA objectives for best results Optical slice thickness = • Small working distances • Flanks up to ~ 70o • Cost ~150K – 250K (FTI IBIS ~1M) • Get a vibration isolation table for your instrument ~7K • Set up in a (dry) basement if possible • Accuracy down to +/- 10 nm Confocal Microscope Trivia • Some manufactures: • Olympus • LEXT (Laser) • Zeiss • CSM (White Light) • LSM (Laser) • Nanofocus • msurf series (White Light) • Sensofar/Leica • Plu series/DCM (White Light) Focus Variation Microscope “Low res” common Focus variation mic ~ +/- 1mm Scherer and Prantl Detector Source Out of focus light In focus light Objective lens Tool mark surface (profile of a striation pattern) Focal plane for objective Sample stage Cutaway Alicona, GMBH Objective’s focal plane Sample stage Scan stage in “z”-direction For Each Detector Pixel: Detector Focus Measure Axial Focus Function Point on surface corresponding to pixel is in maximum focus here z Record the “axial response” as stage is moved along the z-direction Focus Determination: Detector Pixel of interest Compute standard deviation (sd) of pixels grey values in the neighborhood A pixel in focus sits in a neighborhood with a large sd Focus Variation Microscope Trivia • Use high NA objectives for best results • Can use external light • Large working distances • Flanks up to ~75o • Cost ~200K – 250K. • 80K models WON’T have the vertical resolution needed for forensic work • Get a vibration isolation table for your instrument ~7K • Set up in a (dry) basement if possible • Accuracy down to +/- 10nm Focus Variation Microscope Trivia • Some manufactures: • Alicona • IFM • Can get optional rotational stage • Sensofar/Leica • S neox/DCM Interferometer Fixed mirror recombine split Incoming wave Path lengths equal Recombine in-phase Movable mirror Interferometer Fixed mirror recombine split Incoming wave Path lengths NOT equal Recombine out-of-phase Movable mirror Interferometric Height Measurement • The basic idea: • • • • Each surface point is a “fixed mirror” Move a reference mirror in objective Split beams recombine in and out of phase Constructive interference occurs when surface points in focal plane • Infer the surface heights from where constructive interference occurs Interferometric Microscope Wyant Early Interferometric Microscope for Surafce Metrology James Wyant Wyant Early Interferometric Microscope Modern Interferometric Microscope for Surafce Metrology Source Camera (Detector) Microscope Configuration Reference mirror Objective lens Piezo Scan objective for Interference in “z”-direction Beam-splitter Tool mark surface (profile of a striation pattern) Path lengths equal Point in focus Focal plane for objective Sample stage Source Camera (Detector) Microscope Configuration Reference mirror Objective lens Piezo Scan objective for Interference in “z”-direction Beam-splitter Tool mark surface (profile of a striation pattern) Path lengths un-equal Point in out of focus Focal plane for objective Sample stage Interference Objectives Michelson objective ~ 2X – 10X Mirau objective ~ 10X – 100X Linnik objective + 100X For Each Detector Pixel: Detector Intensity Pixel Interference Pattern Point on surface corresponding To pixel’s is in maximum focus here z Record each pixels interference pattern as objective is scanned Inference patterns: Scan objective for Interference in “z”-direction Sample stage Fringes Bruker NSD Fringe Pattern Bruker NSD Surface Turn Fringes Into A Surface Intensity for each detector pixel: Fourier transform I(z) to get q(k) with: arg[q(k)] Compute surface heights A q k0 k deGroot Interferometry Trivia • Use high NA objectives for best results • Small working distances • Flanks up to ~25o • Cost ~200K – 250K. • Get a vibration isolation table for your instrument ~7K • Set up in a (dry) basement if possible • Comes in two modes • VSI: Accuracy +/- 10nm • PSI: Accuracy below 1nm Interferometry Trivia • Some manufactures: • Bruker (Acquired WYKO/Veeco) • Taylor Hobson • Sensofar/Leica • S neox/DCM Surface Data Land Engraved Area: Surface heights (mm) 37.88 37.88 37.86 37.84 37.81 37.81 37.82 37.84 37.84 37.80 37.89 37.89 37.85 37.82 37.80 37.78 37.80 37.81 37.80 37.77 37.89 37.87 37.84 37.81 37.80 37.79 37.80 37.80 37.76 37.76 37.90 37.87 37.84 37.81 37.82 37.82 37.83 37.81 37.77 37.74 37.92 37.87 37.84 37.83 37.84 37.85 37.87 37.84 37.78 37.79 37.91 37.85 37.85 37.85 37.86 37.89 37.91 37.89 37.86 37.84 Point are “double precision”: 64-bits/point 37.93 37.89 37.85 37.88 37.89 37.94 37.98 37.95 37.92 37.90 37.93 37.92 37.92 37.92 37.94 37.96 37.99 37.99 37.96 37.93 37.94 37.97 37.98 37.97 37.98 38.00 38.02 38.01 38.00 37.98 BIG FILES! 37.99 38.02 38.03 38.04 38.05 38.04 38.05 38.06 38.03 38.00 Surface Data Land Engraved Area: Detector levels (16-bit values): 16617 16618 16606 16600 16585 16587 16591 16597 16599 16581 16622 16620 16602 16589 16583 16572 16581 16586 16581 16567 16622 16613 16600 16587 16583 16579 16583 16583 16566 16562 16625 16613 16597 16587 16588 16590 16594 16585 16569 16556 16632 16610 16597 16594 16599 16604 16610 16597 16574 16575 16629 16605 16603 16603 16608 16622 16630 16623 16607 16597 Point are detector grey levels: 16-bits/point 16638 16622 16604 16616 16619 16641 16661 16646 16634 16625 16639 16632 16632 16632 16643 16652 16663 16666 16651 16640 16645 16656 16662 16658 16662 16669 16679 16674 16669 16660 16665 16676 16684 16686 16689 16688 16692 16695 16683 16671 Smaller files. Convert to mm in RAM Surface Data Trivia • Different systems use different storage formats • Be aware if writing custom apps. ASK COMPANY FOR FILE FORMAT! • Alicona: Saves surface data as doubles. HUGE FILES! • Zeiss: Saves surface data as 16-bit grey levels with conversion factor • Other?? 24, 32-bit detectors now?? • Need to standardize file format! • X3DZhang,Brubaker • Digital-Surf .surPetraco Surface Filtering • Think of a toolmark surface as being made up of a series of waves Amplitude (mm) −1.0 0.0 1.0 Sinusoid 1 0 2 4 x (mm) 6 8 Amplitude (mm) −0.4 0.0 0.4 Sinusoid 2 Amplitude (mm) −1.5 −0.5 0.5 1.5 (Fake) Profile = Sinusoid 1 + Sinusoid 2 0 0 2 4 x (mm) 6 8 2 4 x (mm) 6 8 Surface Filtering • Examine different scales by “blocking out” (filtering) some of the sinusoids Filter Transmission Characteristic Amplitude (%) 0 20 60 Amplitude (mm) −1.5 −0.5 0.5 1.5 (Fake) Profile = Sinusoid 1 + Sinusoid 2 0.0 0.4 4 x (mm) −0.4 2 Re(ifft(zf * fs)) 0 6 8 “Low Pass” filter blocks high frequencies and passes low frequencies (long wavelengths) 5e−03 5e−02 5e−01 wavelength (mm) 5e+00 Surface Filtering • Examine different scales by “blocking out” (filtering) some of the sinusoids Filter Transmission Characteristic “High Pass” filter blocks low frequencies and passes high frequencies (short wavelengths) 0 Amplitude (%) 40 80 Amplitude (mm) −1.5 −0.5 0.5 1.5 (Fake) Profile = Sinusoid 1 + Sinusoid 2 6 5e−03 8 5e−02 5e−01 wavelength (mm) 0.0 0.2 4 x (mm) −0.3 2 Re(ifft(zf * fs)) 0 0 2000 4000 6000 8000 5e+00 Surface Filtering Trivia • Wavelength “cutoffs” lcut Amplitude (%) 40 80 A “Low Pass” filter Filter Transmission Characteristic A “High Pass” filter lcut 0 Amplitude (%) 0 20 60 Filter Transmission Characteristic 5e−03 5e−02 5e−01 wavelength (mm) 5e+00 5e−03 • Wavelength ranges • Short wavelengths passed: roughness • Medium wavelengths passed: waviness • Long wavelengths passed: form 5e−02 5e−01 wavelength (mm) 5e+00 Surface Filtering • Band-pass filter: Select narrow wavelength bands to keep. • High-pass/Low-pass combinations (Filter banks) • Wavelets are great at doing this Statistics Weapon Mark Association – What measurement techniques can be used to obtain data for toolmarks? – What statistical methods should be used? • How do we measure a degree of confidence for an association, i.e. a “match”? • What are the identification error rates for different methods of identification? Why ? • R is not a black box! • Codes available for review; totally transparent! • R maintained by a professional group of statisticians, and computational scientists • From very simple to state-of-the-art procedures available • Very good graphics for exhibits and papers • R is extensible (it is a full scripting language) • Coding/syntax similar to MATLAB • Easy to link to C/C++ routines Why ? • Where to get information on R : • R: http://www.r-project.org/ • Just need the base • RStudio: http://rstudio.org/ • A great IDE for R • Work on all platforms • Sometimes slows down performance… • CRAN: http://cran.r-project.org/ • Library repository for R • Click on Search on the left of the website to search for package/info on packages Finding our way around R/RStudio Common Computational Practice • Gauge similarity between tool marks with one number • Similarity “metric” is a function which measures “sameness” • Only requirement: s(A,B) = s(B,A) • There are an INFINITE number of ways to measure similarity!! • Often max CCF is used. Cross-correlation 0.0 1.0 −1.5 z (mm) NIST LEA 1 0 200 400 600 800 1000 1400 −1.5 0.0 1.0 z (mm) NIST LEA 2 x (mm) 0 200 400 600 800 1000 NIST 1 vs. NIST 2 1400 0.0 −0.2 CCF x (mm) −1000 −500 0 Lag 500 1000 Cross-correlation R2 = - 0.04 −1.5 0.0 1.0 Lag = 0 −1.5 −1.0 −0.5 0.5 1.0 R2 = 0.17 −1.5 0.0 1.0 Lag = 45 0.0 −1.5 −1.0 −0.5 0.0 0.5 1.0 −0.5 0.0 0.5 1.0 KNM canRef=Blue, sometimesSliding=Red have high max-ccf… 0 2000 4000 6000 8000 Index max-ccf: 0.751 0.0 0.4 0.8 Ref=Blue, Sliding=Red 0 2000 4000 6000 8000 Cross-Correlation • Glock primer shear: Each profile ~2+ mm • Lag over 2000 units (~0.8 mm) • Max CCF distributions Scores from “Known Matches” Scores from “Known Non-Matches” 2.0 1.5 1.0 0.5 0.0 Density 2.5 3.0 3.5 Max CCF Similarity Score Distributions 0.0 0.2 0.4 0.6 0.8 1.0 Max CCF Score We thought: Ehhhhhh……. Multivariate Feature Vectors • Random variables - All measurements have an associated “randomness” component • Randomness –patternless, unstructured, typical, total ignoranceChaitin, Claude • For an experiment/observation, put many measurements together into a list • Collection random variables into a list called a random vector 1. Also called: observation vectors feature vectors Multivariate Feature Vectors • Potential feature vectors for surface metrology • • • • Entire surfaces *Surface profiles Surface/profile parameters Surface/profile Fourier transform or wavelet coefficients • Translation/rotation/scale invariant surface (image) moments Mean total profile: 0.0 1 -0.5 0 rep(0pr, olefingtlbar[dprerhco(iodedvfier.lzeriv.ozs]eros)) Barcode representation 0.5 2 1.0 Mean waviness profile: -1.0-1 Waviness profile 0 1000 2000 3000 1:length(barcode) 1:length(profil) deriv.zeros 4000 5000 Consecutive Matching Striae (CMS)-Space Toolmarks (screwdriver striation profiles) form database Biasotti-Murdock Dictionary Some Important Terms • Latent Variable: weighted combination of experimental variables into a new “synthetic” variable • Also called: scores, components or factors • The weights are called loadings • Most latent variables we will study are linear combinations between experimental variables and loadings: • Dot prod. between obs. vect. and loading vect. gives a score: z = x · a Principal Component Analysis • PCA: • Is a rotation of reference frame • Gives new PC directions’ relative importance • PC variance Principal Component Analysis • Technically, PCA is an eigenvalue-problem • Diagonalize some version of S or R to get a PCs • Typically SAPC = APCs covariance matrix of PC matrix “loadings” matrix of PC variances • For a data frame of p variables, there are p possible PCs. • s ≅ PC importance, dimension reduction • Scores are data projected into space of PCs retained ZPC = XAPC • Scores plots, either 2D or 3D Setup for Multivariate Analysis • Need a data matrix to do machine learning é X11 .. X1 j ê : ê : ê X = Xi1 .. Xij ê ê : : ê X êë n1 .. X nj .. X1 p ù ú : ú .. Xip ú ú : ú .. X np úú û Represent as a vector of values {-4.62, -4.60, -4.58, ...} • Each profile or surface is a row in the data matrix • Typical length is ~4000 points/profile • PCA can: • 2D surfaces are far longer • Remove much of the redundancy • HIGHLY REDUNDANT representation • Make discrimination computations of surface data far more tractable • 3D PCA of 1740 real and simulated mean profiles of striation patterns from 58 screwdrivers: • How many PCs should we use to represent the data?? • • • ~45% variance retained No unique answer FIRST we need an algorithm to I.D. a toolmark to a tool Support Vector Machines • Support Vector Machines (SVM) determine efficient association rules • In the absence of specific knowledge of probability densities SVM decision boundary Support Vector Machines • SVM computed as optimization of “Lagrange multipliers” 1 max å l - å å l l y y k(x , x ) 0 £ l £ C i 2 n l n n i i =1 i j i j i j i =1 j =1 • Quadratic optimization problem • Convex => SVMs unique unlike NNs • k(xi,xj) kernel function • “Warps” data space and helps to find separations • Many forms depending on application: linear, rbf usually • C: penalty parameter • control the margin of error between groups that are not perfectly separable: 0.1 to 10 usually Support Vector Machines • The SVM decision rule is given as: æ n ö g(xi ) = sign ç å l j y j k(xi , x j ) + b÷ è j =1 ø • Equation for a plane in “kernel space” • Multi group classification handled by “voting” PCA-SVM • How many Principal Components should we use? With 7 PCs, expect ~3% error rate With 13 PCs, expect ~1% error rate Canonical Variate Analysis • This supervised technique is called Linear Discriminant Analysis (LDA) in R • Also called Fisher linear discriminant analysis • CVA is closely related to linear Bayes-Gaussian discriminant analysis • Works on a principle similar to PCA: Look for “interesting directions in data space” • CVA: Find directions in space which best separate groups. • Technically: find directions which maximize ratio of between group to within variation Canonical Variate Analysis Project on PC1: Not necessarily good group separation! Project on CV1: Good group separation! Note: There are #groups -1 or p CVs which ever is smaller Canonical Variate Analysis • Use between-group to within-group covariance matrix, W-1B to find directions of best group separation (CVA loadings, Acv): ( ) W-1B A cv = A cvs • CVA can be used for dimension reduction. • Caution! These “dimensions” are not at right angles (i.e. not orthogonal) • CVA plots can thus be distorted from reality • Always check loading angles! • Caution! CVA will not work well with very correlated data Canonical Variate Analysis • Distance metric used in CVA to assign group i.d. of an unknown data point: arg min éëd ( x ) ùû j =1,...k T j ( d j (xunk ) = -x A cvA T cv )x unk ( grp j unk ) 1 T T + x j A cvA cv x j - ln [ Pr(grp j)] 2 • If data is Gaussian and group covariance structures are the same then CVA classification is the same as Bayes-Gaussian classification. Canonical Variate Analysis • 2D/3D-CVA scores plots of RB screwdrivers 2D CVA 3D CVA PCA vs. CVA • 2D scores plots of RB screwdrivers: 2D PCA of striation pattern mean profiles 2D CVA of striation pattern mean profiles Error Rate Estimation • Discriminant functions are trained on a finite set of data • How much fitting should we do? • What should the model’s dimension be? • Model must be used to identify a piece of evidence (data) it was not trained with. • Accurate estimates for error rates of decision model are critical in forensic science applications. • The simplest is apparent error rate: • Error rate on training set • Lousy estimate, but better than nothing Error Rate Estimation • Cross-Validation: hold-out chunks of data set for testing • Known since 1940s • Most common: Hold-one-out • Bootstrap: Randomly selection of observed data (with replacement) • Known since the 1970s • Can yield confidence intervals around error rate estimate • The Best: Small training set, BIG test set 18D PCA-SVM Primer Shear I.D. Model, 2000 Bootstrap Resamples Refined bootstrapped I.D. error rate for primer shear striation patterns= 0.35% 95% C.I. = [0%, 0.83%] (sample size = 720 real and simulated profiles) How good of a “match” is it? Conformal PredictionVovk • Confidence on a scale of 0%-100% • Testable claim: Long run I.D. errorrate should be the chosen significance level • This is an orthodox “frequentist” approach • Roots in Algorithmic Information Theory • Data should be IID but that’s it Cumulative # of Errors • Can give a judge or jury an easy to understand measure of reliability of classification result 80% confidence 20% error Slope = 0.2 95% confidence 5% error Slope = 0.05 99% confidence 1% error Slope = 0.01 Sequence of Unk Obs Vects How Conformal Prediction works for us • Given a “bag” of obs with known identities and one obs of unknown identityVovk • Estimate how “wrong” labelings are for each observation with a nonconformity score (“wrong-iness”) • For us, one-vs-one SVMs: 1 k ( k -1)/2 ti = å l k - 1 j=1 i, j • Looking at the “wrong-iness” of known observations in the bag: • Does labeling-i for the unknown have an unusual amount of “wrong-iness”??: ppossible-ID = i { # j Î{1,2,..., n}: t j possible-IDi n ³ ttest-pattern i possible-ID } i Î{1,2,..., k I.D.s} • If not: • ppossible-ID ≥ chosen level of significance a i • Put IDi in the (1 - a )*100% confidence interval Conformal Prediction • For 95%-CPT (PCA-SVM) confidence intervals will not contain the correct I.D. 5% of the time in the long run • Straight-forward validation/explanation picture for court Empirical Error Rate: 5.3% Theoretical (Long Run) Error Rate: 5% 14D PCA-SVM Decision Model for screwdriver striation patterns Conformal Prediction Drawbacks • CPT is an interval method • Can (and does) produce multi-label I.D. intervals • A “correct” I.D. is an interval with all labels • Doesn’t happen often in practice… • Empty intervals count as “errors” • Well…, what if the “correct” answer isn’t in the database • An “Open-set” problem which Champod, Gantz and Saunders have pointed out • Must be run in “on-line” mode for LRG Pr(error) £ a • After 500+ I.D. attempts run in “off-line” mode we noticed in practice Pr(error) » a How good of a “match” is it? Efron Empirical Bayes’ • An I.D. is output for each questioned toolmark • This is a computer “match” • What’s the probability it is truly not a “match”? • Similar problem in genomics for detecting disease from microarray data • They use data and Bayes’ theorem to get an estimate No diseasegenomics = Not a true “match”toolmarks Empirical Bayes’ • We use Efron’s machinery for “empirical Bayes’ two-groups model”Efron • Surprisingly simple! • Use binned data to do a Poisson regression • Some notation: • S-, truly no association, Null hypothesis • S+, truly an association, Non-null hypothesis • z, a score derived from a machine learning task to I.D. an unknown pattern with a group • z is a Gaussian random variate for the Null Empirical Bayes’ • From Bayes’ Theorem we can getEfron: Estimated probability of not a true “match” given the algorithms' output z-score associated with its “match” ( ) ( ) ˆ p z | S ˆ S- | z = ˆ SPr Pr fˆ ( z ) ( ) Names: Posterior error probability (PEP)Kall Local false discovery rate (lfdr)Efron • Suggested interpretation for casework: • We agree with Gelaman and ShaliziGelman: “…posterior model probabilities …[are]… useful as tools for prediction and for understanding structure in data, as long as these probabilities are not taken too seriously.” ( ) ˆ S- | z = Estimated “believability” of machine made association 1- Pr Empirical Bayes’ • Bootstrap procedure to get estimate of the KNM distribution of “Platt-scores”Platt,e1071 • Use a “Training” set • Use this to get p-values/z-values on a “Validation” set • Inspired by Storey and Tibshirani’s Null estimation methodStorey • Use SVM to get KM and KNM “Platt-score” distributions • Use a “Validation” set From fit histogram by Efron’s method get: ( fˆ ( z ) “mixture” density ) pˆ z | S- z-density given KNM => Should be Gaussian ( ) ˆ S- Estimate of prior for KNM Pr What’s the point?? We can test the fits to z-score ( ) fˆ ( z ) and pˆ z | S ! Posterior Association Probability: Believability Curve 12D PCA-SVM locfdr fit for Glock primer shear patterns +/- 2 standard errors Poisson (Efron) on test set Bayesian Poisson with intercept on test set Bayesian Poisson on test set Bayesian over-dispersed Poisson with intercept on test set Bayes Factors/Likelihood Ratios • In the “Forensic Bayesian Framework”, the Likelihood Ratio is the measure of the weight of evidence. • LRs are called Bayes Factors by most statistician • LRs give the measure of support the “evidence” lends to the “prosecution hypothesis” vs. the “defense hypothesis” • From Bayes Theorem: LR = ( Pr E | H p Pr ( E | H d ( Pr H p | E ) ) = Pr ( H | E ) = Posterior Odds Prior Odds ) Pr ( H ) d p Pr ( H d ) Bayes Factors/Likelihood Ratios • Once the “fits” for the Empirical Bayes method are obtained, it is easy to compute the corresponding likelihood ratios. o Using the identity: ( ) ˆ S- | z Pr(H p | E) = 1- Pr the likelihood ratio can be computed as: LR(z) = ( ˆ S- | z 1- Pr ˆ S- | z Pr ( ) ˆ (S ) 1 - Pr ˆ (S ) Pr - - ) Bayes Factors/Likelihood Ratios • Using the fit posteriors and priors we can obtain the likelihood ratiosTippett, Ramos Known match LR values Known non-match LR values Empirical Bayes’: Some Things That Bother Me • Need a lot of z-scores • Big data sets in forensic science largely don’t exist • z-scores should be fairly independent • Especially necessary for interval estimates around lfdrEfron • Requires “binning” in arbitrary number of intervals • Also suffers from the “Open-set” problem • Interpretation of the prior probability for this application • Should Pr(S-) be 1 or very close to it? How close? How to Carry Out a “Successful” Research Project The Synergy Between Practitioners and Academia Collaboration • Practitioners: • Think about what questions you want to be able to answer with data BEFORE experimentation • Write down proposed questions/design • Be aware that the questions you want answers too MAY NOT have answers • What can you answer?? • Be aware that a typical research project takes 1-2 years to complete Collaboration • Practitioners: • Research projects are NOT just for interns! • Interns typically need tremendous supervision for scientific/applied statistical research • Take a college course on statistics/experimental design • Rate-my-professor is your friend! • Visit local university/company websites to look for the outside expertise you may need. • Visit the department, go to some seminars Collaboration • Academics/Research consultants: • Be aware practitioners cannot just publish whenever and whatever they want • Long internal review processes! • COMMUNICATION!!!!! • Listen carefully to the needs/questions of collaborating practitioners • Negotiate the project design • What kind of results can be achieved within a reasonable amount of time? • Hold regular face to face meetings if possible Collaboration • Academics/Research consultants: • Applied research is not just for undergraduates/high-school interns! • Visit the crime lab!!!!! • Watch the practitioners do their job. • Learn the tools they use day to day! • Microscopy!!!!! • Use their accumulated experience to help guide your design/desired outcomes • What do they focus on?? Fire Debris Analysis Casework • Liquid gasoline samples recovered during investigation: • Unknown history • Subjected to various real world conditions. • If an individual sample can be discriminated from the larger group, this can be of forensic interest. • Gas-Chromatography Commonly Used to ID gas. • Peak comparisons of chromatograms difficult and time consuming. • Does “eye-balling” satisfy Daubert, or even Frye .....???? • 2D PCA • 97.3% variance retained • Avg. LDA HOO correct classification rate: 83% PC 2 0.5 6 6 66 6 6 0.4 6 0.3 4 19 1 18 18 0.2 1 9 9 9 1 1 4 18 4 4 -2.5 5 5 5 5 55 4 4 4 15 16 15 158 1616 8 11 11 11 1 0.1 1 19 5 14 14 7 17 17 10 7 7 7 7 -1.5 10 7 12 12 88 12 2 22 333 3 3 3 2 3 2 2 -1 -0.1 -0.2 10 2 7 1 13 13 13 8 8 20 20 20 19 14 17 8 PC 1 • 2D CVA • Avg. LDA HOO correct classification rate: 92% CV 2 55 5 55 5 0.04 0.02 -0.1 1818 -0.08 -0.06 18 13 13 -0.0413 CV 1 -0.02 16 1616 14 14 14 11 11 11 22 2222 444444 8 88888 4 6 666 9 6 96 9 1 11 11 1 77 77 77 1010 10 33 3 33 3 2019 20 19 20 19 -0.02 0.02 15 15 17 15 17 17 -0.04 -0.06 12 12 12 Accidental Patterns on Footwear • Shoe prints contain marks and patterns due to various circumstances that can be used to distinguish one shoe print from another. • How reliable are the accidental patterns for identifying particular shoes? Facial Recognition Approach to Accidental Pattern Identification 5 y axis 6 0 66 6 6 6 8 6X 8 8 8 8 8 8 -5 X 4 2 4 4 X 0 4 4 4 4 z axis 4 X 9 99 9 9 4 -2 4 -4 3D PCA 59.7% of variance -6 -7.5 -5 -2.5 x axis 0 9 66688 4888 277 6896 11 9 55 31 73 X775 1 7 77 5 7 3X X3 777 33 33 3 X55 5 2 1 2 X 5 5 11 55 11 2 22 2 2 Tool marks • Like shoes, tools can leave marks which can be used in identification • Class characteristics • Subclass characteristics • Individual characteristics Standard Striation Patterns Made with ¼’’ Slotted Screwdriver Measure lines and grooves with ImageJ Translate ImageJ data to a feature vector that can be processed LEA Striations A, 2, #2 Bromberg, Lucky C, 8, #4 Bromberg, Lucky Questioned Documents: Photocopier Identification • Mordente, Gestring, Tytell 3000 2500 2000 1500 1000 500 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 3000 2500 2000 1500 1000 500 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 Photocopy of a blank sheet of paper 3000 2500 2000 1500 1000 500 0500 1000 1500 2000 2500 0500 1000 1500 2000 2500 0500 1000 1500 2000 2500 Dust: Where does it come from? Our Environments! Any matter or substance: • both natural and synthetic • reduces into minute bits, pieces, smears, and residues • encountered as trace aggregates Evidence! N. Petraco Where can you find it? House Work Everywhere Outdoors Vehicle N. Petraco Analyze Results 3D PCA-Clustering can show potential for discrimination Bayes Net for Dust in Authentication Case References • Bolton-King, Evans, Smith, Painter, Allsop, Cranton. AFTE J 42(1),23 2010 • Artigas. In: Optical Measurement of Surface Topography. Leach ed. Springer, 201l • Helmli. In: Optical Measurement of Surface Topography. Leach ed. Springer, 2011 • deGroot. In: Optical Measurement of Surface Topography. Leach ed. Springer, 201l • Efron, B. (2010). Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction. New York: Cambridge University Press. • Gambino C., McLaughlin P., Kuo L., Kammerman F., Shenkin S., Diaczuk P., Petraco N., Hamby J. and Petraco N.D.K., “Forensic Surface Metrology: Tool Mark Evidence", Scanning 27(1-3), 1-7 (2011). • JAGS “A program for analysis of Bayesian hierarchical models using Markov Chain Monte Carlo simulation”, Version 3.3.0. http://mcmc-jags.sourceforge.net/ • Kall L., Storey J. D., MacCross M. J. and Noble W. S. (2008). Posterior error probabilities and false discovery rates: two sides of the same coin. J Proteome Research, 7(1), 40-44. References • locfdr R package. 2011. locfdr: “Computation of local false discovery rates”, Version 1.17.http://cran.r-project.org/web/packages/locfdr/index.html • Moran B., "A Report on the AFTE Theory of Identification and Range of Conclusions for Tool Mark Identification and Resulting Approaches To Casework," AFTE Journal, Vol. 34, No. 2, 2002, pp. 227-35. Petracoa N. D. K., Chan H., De Forest P. R., Diaczuk P., Gambino C., Hamby J., Kammerman F., Kammrath B. W., Kubic T. A., Kuo L., Mc Laughlin P., Petillo G., Petraco N., Phelps E., Pizzola P. A., Purcell D. K. and Shenkin P. “Final Report: Application of Machine Learning to Toolmarks: Statistically Based Methods for Impression Pattern Comparisons”. National Institute of Justice, Grant Report: 2009-DN-BX-K041; 2012. Petraco N. D. K., Kuo L., Chan H., Phelps E., Gambino C., McLaughlin P., Kammerman F., Diaczuk P., Shenkin P., Petraco N. and Hamby J. “Estimates of Striation Pattern Identification Error Rates by Algorithmic Methods”, AFTE J., In Press, 2013. Petraco N. D. K., Zoon P., Baiker M., Kammerman F., Gambino C. “Stochastic and Deterministic Striation Pattern Simulation”. In preparation 2013. Platt J. C. “Probabilities for SV Machines”. In: Advances in Large Margin Classifiers Eds: Smola A. J., Bartlett P., Scholkopf B., and Schuurmans D. MIT Press, 2000. Plummer M. “JAGS: A Program for Analysis of Bayesian Graphical Models Using Gibbs Sampling”, Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003), March 20– 22, Vienna, Austria. Stan Development Team. 2013. “Stan: A C++ Library for Probability and Sampling”, Version 1.3. http://mcstan.org/ Storey J. D. and Tibshirani R. “Statistical significance for genome wide studies”. PNAS 2003;100(16):94409445. Vovk V., Gammerman A., and Shafer G. (2005). Algorithmic learning in a random world. 1st ed. Springer, New York. • • • • • • • • References 20. Tippett CF, Emerson VJ, Fereday MJ, Lawton F, Richardson A, Jones LT, Lampert SM., “The Evidential Value of the Comparison of Paint Flakes from Sources other than Vehicles”, J Forensic Soc Soc 1968;8(2-3):61-65. 21. Ramos D, Gonzalez-Rodriguez J, Zadora G, Aitken C. “Information-Theoretical Assessment of the Performance of Likelihood Ratio Computation Methods”, J Forensic Sci 2013;58(6):1503-1518. Acknowledgements • Professor Chris Saunders (SDSU) • Professor Christoph Champod (Lausanne) • Alan Zheng (NIST) • Research Team: • Dr. Martin Baiker • • Ms. Helen Chan • • Ms. Julie Cohen • • Mr. Peter Diaczuk • • Dr. Peter De Forest • • Mr. Antonio Del Valle • • Ms. Carol Gambino • • • Dr. James Hamby • Ms. Alison Hartwell, Esq. • • Dr. Thomas Kubic, Esq. • Ms. Loretta Kuo • Ms. Frani Kammerman • Dr. Brooke Kammrath • Mr. Chris Lucky • Off. Patrick McLaughlin • Dr. Linton Mohammed • Mr. John Murdock • Mr. Nicholas Petraco • Dr. Dale Purcel • Ms. Stephanie Pollut Dr. Peter Pizzola Dr. Graham Rankin Dr. Jacqueline Speir Dr. Peter Shenkin Mr. Chris Singh Mr. Peter Tytell Mr. Todd Weller Ms. Elizabeth Willie Dr. Peter Zoon Website: Data, codes, reprints and preprints: toolmarkstatistics.no-ip.org/ npetraco@gmail.com