Ready or Not, Here it Comes….
Why It’s Likely that You’ll be Employing Predictive
Coding Technology In 2013
Texas Lawyer In-House Counsel Summit
April 4, 2013
Privileged & Confidential
© DiscoverReady 2013
Introductions
Moderator: Amy Hinzmann, SVP, DiscoverReady LLC
Panel:
•
Thomas Gottsegen, Lead Legal Counsel, Statoil
•
Jim Wagner, CEO, DiscoverReady LLC
Privileged & Confidential
© DiscoverReady 2013
Agenda and Goals
•
What is predictive coding?
•
Key terms and concepts you need to know
•
Case law overview
•
Potential predictive coding use cases
•
Issues most likely to be negotiated with opposing counsel
•
Steps you should take to be prepared for your first predictive coding matter
Privileged & Confidential
© DiscoverReady 2013
Warning
This Next Slide is Going to Freak You Out…..
© DiscoverReady 2013
Privileged & Confidential
Welcome to the World of Predictive Coding….
F-score (Fβ) is a “simple” calculation of how well your predictive coding is
performing calculated as
(1 + β2) x Precision x Recall ÷ (β2 x Precision + Recall)
where the Beta refers to the relative weighting of precision and recall,
and a Beta of one weights the value of precision and recall equally.
Privileged & Confidential
© DiscoverReady 2013
What is Predictive Coding?
Many variations and processes, but the general principle in
layman’s terms is …
“A combination of human and
automated processes in which
humans review a subset of
documents, the computer
analyzes those documents and
decisions, and then the computer
analyzes the remaining set of
documents and recommends
decisions for them.”
Privileged & Confidential
© DiscoverReady 2013
Where We Currently Stand
•
Single most discussed topic in the industry
•
At least 6 cases authorizing or discussing use of predictive coding
•
Regulatory agencies actively pursuing predictive coding solution
•
Abundance of scholarly commentary:
– Grossman/Cormack Study
– Significant discussion by Software Providers (Equivio / Relativity /
Recommind)
– 15 Sessions at LegalTech 2013 primarily focused on predictive
coding
•
Most formal commentary and “buzz” focuses on technology as
substitute for human review
•
Use cases suggest implementations focus on “Predictive Culling”™ in
lieu of search terms, with manual review prior to production
•
No standards or safe harbors
•
Not necessarily a cost saver
Privileged & Confidential
© DiscoverReady 2013
Key Terms and Concepts You Need to Know
•
Richness (prevalence of relevant data)
•
Precision
•
Recall
•
Confidence level
•
F-score
•
Sample size
•
Margin of error
•
Control set
•
Training set
Privileged & Confidential
© DiscoverReady 2013
Understanding the Importance of Richness
•
The prevalence of likely relevant documents (or richness) of a data set impacts
every aspect of a predictive coding process
– Efficacy of overall process
– Training
– Statistical Validation
– Cost
• Low Richness = More Review = Higher Costs
• Strategies for identifying and overcoming low richness
– Statistical sampling to determine richness as part of “evaluation” process
– “Seeding” to increase richness
– Custodian selection
– Identification and use of exemplar documents
•
What should you disclose regarding your seeding efforts?
Privileged & Confidential
© DiscoverReady 2013
Sample Size Not Impacted by Total Population
Sample Size for 30% Richness at Selected Population
Size and 5% Confidence Interval
324
323
322
322
323
323
323
323
321
320
Sample Size
323
319
318
316
314
313
312
5%
310
308
Population Size
Privileged & Confidential
Practical Examples of the Importance of Richness
•
Sampling a randomly selected group of documents at the outset of a matter
can provide valuable information regarding the potential efficacy of a party’s
predictive coding efforts.
•
However, the volume of data that must be sampled to gain an adequate level
of confidence in the richness of the data set depends on both the volume of
likely relevant documents in the data set and the desired margin of error
associated with the results
95% Confidence Level
Richness
5%
10%
20%
30%
40%
50%
70%
10%
18
35
61
81
92
96
81
Margin of Error
5%
2%
73
456
138
864
246
1,537
323
2,017
369
2,305
384
2,401
323
2,017
1%
1,096
3,457
6,147
8,067
9,220
9,604
8,067
Privileged & Confidential
© DiscoverReady 2013
Practical Examples of the Importance of Richness
•
Similarly, a party will statistically sample the “results” of their predictive coding
efforts to determine the “precision” and “recall” of the predictive coding
process
•
Again, the volume of data that must be sampled to validate the precision and
recall of the predictive coding efforts turn on the richness of the data set and
the desired margin of error.
Estimated Recall of 80%
Estimated Precision
65%
Margin ofofError
Richness
5%
10%
20%
30%
40%
50%
70%
10%
1,880
940
470
313
235
166
134
5%
6,820
3,410
1,705
1,137
853
682
487
2%
42,280
21,140
10,570
7,047
5,285
4,228
3,020
1%
163,760
81,880
40,940
27,293
20,470
16,376
11,697
Privileged & Confidential
© DiscoverReady 2013
Distinguishing Relevance vs. Responsiveness
•
In the context of Predictive Culling™ (using predictive coding to cull data in
lieu of search terms), it’s critical to distinguish between “likely relevance” for
culling and “responsiveness” to discovery requests
– Parties frequently take a broad view of relevance for purposes of “training
the system” and identifying the data set that will be subject to manual
review by producing party
– The producing party often takes a much narrower view of responsiveness
for purposes of review and production, focusing on responsiveness to
specific discovery requests rather than general concept of relevance
– It is critical that both parties understand that only a subset of the
documents identified as likely relevant during culling actually will be
produced as responsive documents
Privileged & Confidential
© DiscoverReady 2013
What Have the Courts Said?
•
Da Silva Moore v. Publicis Groupe (11 Civ. 1279 S.D.N.Y)
– Parties agreed to use of predictive coding technology to cull data in lieu of search
terms
– Dispute over implementation
•
EORHB, Inc. v. HOA Holdings, LLC (No 7409 Del. Chancery Court)
– Court issued sua sponte order dictating that parties would use predictive coding
even though neither party requested it
– Also ordered parties to agree on single vendor to house documents
•
Global AeroSpace, Inc. et al. v Landow Aviation, LP (CL 61040 Loudoun County
VA Circuit Court)
– Virginia State Court Judge ordered parties to employ predictive coding over
objections of Plaintiff
– According to court and counsel, technology successfully deployed resulting in cost
savings and increased efficiency
Privileged & Confidential
© DiscoverReady 2013
What Have the Courts Said?
•
In Re Actos (Pioglitazone) Products Liability Litigation (MDL No. 6:11-md-2299,
W.D. La.)
– Parties reached agreement on joint e-discovery protocol utilizing predictive coding
to cull data
•
Kleen Products, LLC et al. v. Packaging Corporation of America, et al. (No. 105711 N.D. Ill)
– Plaintiffs requested that Court order Defendants to use predictive coding technology
to cull data following substantial productions based on search term based culling
– Ultimately ceded to Judge Nolan’s “suggestion” that parties develop acceptable
keyword search strategy
•
Gabriel Technologies Corp et al. v, Qualcomm Inc., et al. (Case No 08CV1992 S.D.
Cal)
– In landmark cost shifting case, court ordered Plaintiffs to reimburse defendants—
among total fees of $12 million—$2.82 million for “computer assisted algorithm
driven document review” of 12 million records
– Also awarded $400,000 for contract attorney costs for reviewing resultant data
– While award was based on bad faith of plaintiff, demonstrates potential costs of
predictive coding solutions
Privileged & Confidential
© DiscoverReady 2013
Potential Use Cases….
Approach
In lieu of
keyword
searching
Pros



Cons
Keyword searching frequently grossly inexact
Generates higher precision and higher recall rates
Results can be used for review prioritization reduces review
of irrelevant data






Review
Prioritization
In lieu of human
review (post
culling)
Inbound
productions
Produce all to
regulatory
agency




Accelerated access to most relevant data
Ability to direct most relevant data to specific reviewers
Defer review of irrelevant data
Potential use for early case assessment





More consistent decisions
Potentially eliminate review of irrelevant files (low scores)
Significantly reduce costs



Rapid access to most relevant data
Potential to use outbound data set and decisions for training
purposes
Defer or eliminate review of low relevance files
Allows regulators complete access to data set
Particularly beneficial when there is low confidence in
efficacy of keyword searching










Privileged & Confidential
© DiscoverReady 2013
Technology likely costs substantially more than keyword searching
Training can take more time than creation of keyword searches
Can generate more documents for review or production than
keyword searching (not necessarily a negative)
Complexity of negotiating cut-off point with opposing counsel
Must be prepared to argue proportionality in terms of tradeoff of
precision and recall
All documents above cutoff point still have to be reviewed
(identical to keyword searching)
Significant technology costs
Typically defers review, but does not eliminate review of less
relevant data
Higher “relevance” score does not necessarily translate to “hot
doc”
Risk of production of data that has not been reviewed (privilege,
privacy and confidentiality)
Opposing counsel may not approve use of predictive coding
decision of “irrelevant” if the document contains a keyword that
has been stipulated
Technology expense
May not find “hot documents”
Requires privilege, privacy and confidentiality screening of entire
data set
Many clients require review of any document produced prior to
production
Harder to find same documents that regulators may identify as
important
Comparing Your Options
Approach
In lieu of
keyword
searching
Risk
Low
Potential for Cost Savings
Mixed
Should yield better results than many
keyword searches
Can measure efficacy (precision / recall)
Technology costs could be higher
May lower cost by eliminating review of
irrelevant data
May increase cost by generating more
files in total requiring review
Lowest
Review
Prioritization
Mixed
Absent combination with another
approach, producing party will still review
entire data set for relevance, privilege,
and confidentiality prior to production
Technology costs could be higher
May lead to hot docs quickly
Typically only defers review, does not
eliminate the review of irrelevant files
High
In lieu of
human review
(post culling)
High
Risk of production of data that has not
been reviewed (privilege, privacy and
confidentiality)
Cost savings of 50%-75% (or more) for
every computer decision vs. human
decision
Low
Inbound
productions
High
No worries of inadvertent productions or
material omissions
Possible to miss hot docs
Can significantly reduce volume of data
reviewed
Client can make risk/reward decision of
not reviewing low ranking documents
High
Produce all to
regulatory
agency
None
Could be difficult to identify same
documents as regulators identify
Risk of production of privileged, private
or confidential info
Creates burden to analyze and review
entire data set
Privileged & Confidential
© DiscoverReady 2013
Prevalence
High
Most common public commentary is
inadequacy of poorly constructed search
terms
Predictive coding will likely beat keyword
searches “head to head” and could
become a default standard for culling data
Highest
Very similar to traditional methods of
“clustering” review set based on likely
similarity between the documents
Accepted process that provides increased
accuracy and efficiency
Very low
Clients uncomfortable producing data that
has not been reviewed
Medium
Represents a “baby step” into predictive
coding
Low (but increasing)
Traditionally regulatory agencies have
avoided predictive coding
Likely increasing as evidenced by SEC’s
recent agreement to bring Recommind in
house
Issues Most Likely to be Negotiated
•
Degree of collaboration/transparency
•
Software
•
Pre-training data culling (de-dupe/de-Nist/etc.)
•
Training methodology (richness of data directly impacts training time
and cost)
•
Addressing document families (analysis is usually on individual
documents)
•
Who reviews the training set
•
Privilege, privacy and confidentially screening
•
Negotiating a cut-off point for irrelevant data
•
Negotiating acceptable margin of error and confidence level
•
Production format
Privileged & Confidential
© DiscoverReady 2013
3 Places Where Actos Went Wrong
1. Use of 4 custodians for training set
– Documents possessed by the selected custodians may not be representative of the
documents possessed by the larger custodian group, making training at best incomplete
and at worst inappropriate for the larger data set
2. Establishing target confidence levels and margin of error without any knowledge
regarding potential richness of data set
– The volume of data required to be reviewed to establish specific confidence levels and
margins of error will depend in large part on the richness of the data set.
3. Collaborative joint training
– This is certainly the largest issue with the Actos protocol, as it would require 3 subject
matter experts from each party to review a control set, training set, and samples of the
resulting work product.
– In addition to providing unprecedented levels of participation in the discovery process,
likely will drive large inefficiencies and drive significant costs associated with training
process both because of the extensive level of collaboration required and the lack of
specific protocols to resolve disputes among the subject matter experts
Privileged & Confidential
© DiscoverReady 2013
Options For Your Negotiations
Actos
Training
methodology
Who reviews the
training set
DiscoverReady Best Practice
Global Aerospace—OrcaTech
Da Silva Moore—Recommind
Predictive coding should be more about process than
software. As such, the selection of software will
depend on your goals and objectives in a particular
matter
Data population de-duplicated and deNisted with spam, commercial email
(blasts) and no-text files removed
Global Aerospace—de-duplication and deNisting only
De-duplication and de-Nisting, with evaluation of
removing commercial email blasts, email highly unlikely
to be responsive based on sender domain, and no-text
files
Training to occur on major issues only
(Responsive/Non-Responsive/Skip)
Equivio—in addition to major categories,
contemplates training on individual issues
to organize review
Grossman—contemplates training on both
major issues and four individual issues
Training on major issues with review prioritization based
on likely relevance.
Software
Data Culling
Other Approaches
Equivio Relevance
Collaborative, which Actos defines as
having 3 representatives from each party
review all training documents and
achieve consensus across each
document in training set
Global Aerospace—producing party to train
individually and then provide entire training
set (absent privileged and sensitive NR
documents) to plaintiff
Da Silva—producing party to train using
keyword hits (based on PC tool) and
provide hit counts and documents
containing hits to plaintiff
Equivio—collaborative, which is defined as
having 3 reviewers from producing party
review and reach consensus on each
document
Grossman—Individual decisions with single
“gold standard” assessor for QC
Privileged & Confidential
© DiscoverReady 2013
Must distinguish between relevance for purposes of
culling and relevance for purposes of production
Depending on the richness of the data set and the
scope of the responsiveness determinations,
consideration should be given to potentially “seeding”
the data set to include a higher volume of responsive
documents than might otherwise appear across the data
set (hopefully diminishing the training required due to
low richness an very targeted responsiveness criteria)
Producing party trains using individual subject matter
experts with QC by “gold standard” assessor.
Plaintiff’s validation of training to be accomplished by
production of precision and recall scores, and samples
of documents defined to be non-responsive beneath
proposed data cutoff. A producing party should resist
requests from the opposing party to participate in
training or otherwise be allowed to review the entirety of
the training sets to avoid broad interpretation of
relevance beyond that established by the producing
party unless having similar access to the opposing
party’s data likely would prove beneficial.
Options For Your Negotiations
Actos
Not addressed
Other Approaches
Not addressed
Addressing
Document Families
DiscoverReady Best Practice
Training is performed on individual documents without
taking family members into account for purposes of
determining relevance.
For any document with a sufficiently high relevance
score, all members of the document’s family will be
included in the attorney review prior to production.
Documents will be produced, and withheld (as
appropriate) as families.
Negotiating Cutoff
Point for Irrelevant
Data
Collaborative approach, with plaintiffs
allowed to statistically sufficient sample
set of documents determined to fall
below proposed cutoff
Global Aerospace and Da Silva similarly
advocate collaborative approach built
around representative sampling of data
below proposed cut-off
Follow Actos and agree to collaborative approach based
on representative sampling of data below proposed cutoff
Consider attempting to reach agreement on floor or
standard at which cutoff will occur (i.e. not agreeing to a
specific “score” but instead agreeing that cutoff will
occur no lower than at a point at which the percentage of
documents likely to be responsive falls below a certain
threshold regardless of the score at which that occurs)
Negotiating
Acceptable Margin
of Error /
Confidence Level
Collaborative approach based on
identification of control sets prior to
review. Parties agreed to 95%
confidence level and +/- 5% margin of
error
Global Aerospace-train to 75% recall level
Da Silva—Iterative training until change in
total number or relevant documents
predicted by the system is less than 5% as
compared to the prior iteration and no new
documents are found that are predicted to
be hot
Privileged & Confidential
© DiscoverReady 2013
Follow Actos in spirit, but make no commitment to
specific margin of error and confidence until more is
known about the data set for purposes of identifying
level of review required to achieve desired levels in light
of richness of data population. Objective is to avoid
necessity of sampling inordinate volume of data.
Other Considerations
•
Selecting your predictive coding software
•
Know that opposing counsel is going to actively participate in your discovery
process to a level uncommon for manual review
•
Use Actos protocol as a stylistic guide to set out SI protocol to define and set
out:
– Data sources and custodians subject to predictive coding process
– Process to be followed (and tight dispute resolution provisions)
– Effective claw back agreement
•
Utilize automated privilege screening process or other well defined quality
privilege screening process for training sets and sample sets
•
Document your process and have an audit trail for all “predicted” decisions
•
Mocking your 30(b)(6) and your pre-trial conferences
• Understand whether you are achieving your goals taking into account
technology, consulting, project management and all forms of review
Privileged & Confidential
© DiscoverReady 2013
Discover Better.
DiscoverReady.
Thank you.
Privileged & Confidential
How does it work? Basic predictive coding
Experts review a sample “training set” and the predictive coding
tool “scores” the remaining documents in the data set based
upon likely relevance
Example
Predictive
Coding Score
1.00
Likely Relevant
0.70
0.69
Training Set
Indeterminate
0.30
Data Set
0.29
Number of documents
required for training
depends on a) richness
of the data and b)
desired confidence
level
Privileged & Confidential
© DiscoverReady 2013
Likely Not Relevant
0.00