Geoff Black
Director, High Tech Investigations
Prudential
The views expressed in this presentation are solely those of the presenter and do not necessarily reflect the views of the presenter’s employer.
Recommended Reading
Why do we need Statistics?
(Ensuring Quality in eDiscovery)
Professional standards
Savvy judges already require sampling
Defensibility
Types of Sampling
Judgmental
Statistical*
A Recent Experience with Sampling
Setting the stage
A Recent Experience with Sampling
The Challenge
Select appropriate filters for a large data set
Audit reviewers without double reviewing everything
Test our processing tools
Accomplish all of these with a high confidence level and
low confidence interval
Statistics for eDiscovery
Confidence Interval
The “confidence interval” or margin of error
How closely our results will reflect the
general population
Lower is better
Statistics for eDiscovery
Confidence Interval Example
We have 100 documents and our
confidence interval is ± 2%.
Testing shows 10% responsiveness
General population should show between
8% and 12% responsiveness, or
8 to 12 documents.
Statistics for eDiscovery
Confidence Level
The “confidence level”
Does our sample accurately represent
the results of general population?
Higher is better
Statistics for eDiscovery
Sample Sizes for Population of 1,000,000
4,500
4,000
3,500
3,000
2,500
99% Confidence Level
2,000
95% Confidence Level
90% Confidence Level
1,500
1,000
500
0
± 10%
± 5%
Margin of Error
± 2%
[Scaling] Statistics for eDiscovery
Sample Sizes at 99% Confidence ± 2%
4,400
4,200
4,000
3,800
3,600
3,400
3,200
3,000
2,800
10,000
100,000
Population Size
1,000,000
10,000,000
A Recent Experience with Sampling
Filtering Selection
Finding a good search method is difficult
Who chooses search terms?
Requires iterative testing and validation
A Recent Experience with Sampling
Validating Filters
Began with around 10,000,000 documents
A 99% confidence level with a ± 2% confidence interval
dictated a sample size of 4,150 documents
Chose a random sample and searched
Reviewed all the results (positive and negative)
A Recent Experience with Sampling
Validating Filters
Results did not match expectations
Revised the list of search terms
Tested the filtering again, and…
A more accurate search with less responsive data!
A Recent Experience with Sampling
Validating Filters
Wait a minute, I always test my keywords!
Not whether you test, but how much data…
A Recent Experience with Sampling
Validating Review
After filtering about 120,000 documents to review
Reviewers often disagree about relevance or
simply don’t understand the material
Double and triple review kills budgets
Instead, sample a random set of 4,010 reviewed documents
A Recent Experience with Sampling
Validating Review
Subject matter expert noted a few anomalies
Re-reviewed items with the confusing term
One reviewer’s results could not be trusted
A Recent Experience with Sampling
Keeping Your Vendors Honest
How do they test their tools?
How were automated tools used in your matter?
Do you know what they cannot do?
How did you use the results in your decisions?
What’s Next?
Built-in iterative review with statistical sampling
Relying solely on “Concept Searching”
is a black box and a dead end
Advanced search techniques must offer
explanatory reasoning
What does all this mean?
(The Benefits of Using Statistics)
Small dataset for testing
Minimize false positives
More accurate search, reduced data volume
Defensibility of statistically validated testing
One last thing…
Technologies will always differ and change rapidly,
but statistical validation is a timeless truth.
References & Related Cases
— The Sedona Conference Working Group Series, “Commentary
on Achieving Quality in the E-Discovery Process,” May 2009.
— Losey, Ralph. “The Multi-Modal ‘Where’s Waldo?’ Approach to
Search…,” 2010. http://e-discoveryteam.com/2010/02/27/
— William A. Gross Construction Associates, Inc. v. American
Manufacturers Mutual Insurance Co., 256 F.R.D. 134, 134
(S.D.N.Y. 2009)
— Victor Stanley v. Creative Pipe, 250 F.R.D. 251 (D. Md. 2008)
— In re Seroquel Products Liability Litigation, 244 F.R.D. 650, 662
(M.D. Fla. 2007)
Geoff Black
geoff@geoffblack.com
www.geoffblack.com/ediscovery