Anti-Fraud Analytics and
Electronic Discovery:
Bridging the gap between Legal, IT and Audit
IIA and ISACA Orange County Chapters
March 11, 2013
Matthew L. Miller, Esq.
Fraud – An Expensive Issue
Barclays was fined £290m by for its role in the Libor fixing scandal
(The Guardian, August 29, 2012)
GlaxoSmithKline to pay $3 Billion in fines for criminal and civil
violations involving 10 drugs
(NY Daily News, July 3, 2012)
US Expands Probe Of Lilly's Compliance With Anti-bribery Law
(The Wall Street Journal, February 23,2010)
Alcatel to Pay $137 Million to Avoid US Prosecution
(Business Week, February 18, 2010)
Stanford convicted and sent to jail for $7 Billion fraud case
(Boston Globe, June 29, 2009)
Halliburton to Pay $559 Million To Settle Bribery Investigation
(The Wall Street Journal, January 27, 2009)
Bribery Case Will Cost Siemens $1.6 Billion
(The New York Times, December 16, 2008)
Page 2
IIA / ISACA Orange County Chapters
Current environment
The perfect storm for fraud & business corruption
Companies are
downsizing and
decentralized
which has an
immediate effect
on internal
controls
Page 3
Internal and External
Pressure
Large
construction and
vendor project
relationships
Stressed and
disaffected
employees may
have greater
ability to
rationalize
improper actions
Opportunity to
Commit Fraud
Market levels
remain low
Regulatory focus
increased at
State and Federal
level
Budgets are
decreasing.
Companies and
organizations are
doing more with
less.
Opportunity
Internal
Controls
Pressure
Rationalization
IIA / ISACA Orange County Chapters
The Fraud Tree – Risk areas
Asset Misappropriation
►
►
►
►
►
►
►
►
►
►
►
Corruption / FCPA
Cash Disbursements
►
Cash Larceny
►
Fake Vendors
►
General Ledger
►
Materials Management &
►
Inventory Control
►
Purchase Order
►
Management
►
Salaries & Payroll Fraud
►
Theft of Assets – Inventory/ ►
Accounts Receivable / Fixed
►
Assets
Travel & Expenses Fraud ►
►
Vendor Management
Payment Cards
Page 4
Bid-Rigging
Bribery
Charity Payments
Conflicts of Interest
Contract Compliance
Customs Payments
Gifts & Entertainment
Government Customers
Illegal Gratuities
Kickbacks
Payments to Agents
Petty Cash
Procurement Rigging
IIA / ISACA Orange County Chapters
Financial Misstatement
►
►
►
►
►
►
►
►
►
Accounts Payable
Account Receivable
Deposits
GAAP
General Ledger
Non-financial
Reserves Analysis
Revenue Recognition
Sales Analysis
What is impact for the three types of Fraud
Types of Fraud
Category
Description
Examples
% of all
cases*
Average
Loss
Asset
Misappropriation
Schemes in which the
perpetrator steals or misuses
an organization’s resources.
►
Fraudulent invoicing
► Payroll fraud
► Skimming cash receipts
►Forging company checks
86.7%
$120,000
Corruption
Schemes involve the
employee’s use of their
influence in a way that violates
his or her duty to the employer
for themselves or someone
else.
►
Accepting or paying a
bribe
►Extortion
►Undisclosed
conflict of interest
33.4%
$250,000
Fraudulent
Statement Fraud
Schemes involving the
intentional misstatement or
omission of material information
in the financial reports.
►
Booking fictitious sales
► Recording expenses in
the wrong period
► Concealing liabilities
7.6%
$1,000,000
Source: ACFE Report to the Nation 2012
*sum of %s in this chart exceeds 100% because several cases involved schemes from more than one category
Page 5
IIA / ISACA Orange County Chapters
http://www.transparency.org/cpi2012/results
Page 6
IIA / ISACA Orange County Chapters
Components of an effective anti-fraud and
compliance program
Assess
Improve
Codeof
Ethics
Corruption&
Fraud
Prevention
Policies
Proactive
Communication
and
Training
Reactive
Analytics/
Controls/
Monitoring
Risk
Assessment
Monitor
SettingtheProperTone
Incident
Response
Plan
Communicate
Action
Action
Action
Action
Action
Action
Action
Develop
Adetailed
Anti-corruptionwork
program
Raise
awarenessand
understandingof
corruptionrisksand
laws
Conduct
aCorruptionrisk
assessment
ContinuousMonitor
effectivenessof
Corruptioncontrols
throughanalytics
Review resultsand
developaction
plantoaddress
FCPA/corruption
issues
Respond
toincidentsand
allegationsofbribery
andcorruption
Communicate
ToStakeholders
Page 7
IIA / ISACA Orange County Chapters
How is fraud detected?
50.3 % by
tip or
accident!
*ACFE 2008 Report to the Nation On Occupational Fraud
Source: ACFE Report to the Nation 2012
Page 8
IIA / ISACA Orange County Chapters
Who is reporting fraud?
*ACFE 2008 Report to the Nation On Occupational Fraud
Source: ACFE Report to the Nation 2012
Page 9
IIA / ISACA Orange County Chapters
Page 10
IIA / ISACA Orange County Chapters
Is your sample seeing the picture?
Page 11
IIA / ISACA Orange County Chapters
Data sources – Selected examples
►
Unstructured Data
►
►
►
►
►
►
►
Email
Instant Messages
Text/Mobile Device Messages
Phone Records
Social Media
Trade Press and Commentary
Structured Data
►
►
►
►
►
►
►
Page 12
Financial Records
T&E
Claims Data
Purchase Orders
Inventory Records
Employee and Vendor Lists
Public Databases
IIA / ISACA Orange County Chapters
Data Mining: Techniques
►
Unstructured
►
►
►
►
►
►
►
►
Structured Data
►
►
►
►
►
Emotive Tone Analysis
Document Classification
Topic Modeling
Concept Induction
Fact Pattern Analysis
Social Network and Actor Analysis
Predictive Modeling
Fraud Scenario Tests
Temporal Analysis
Anomaly Detection
Cluster Analysis
Integrated
►
►
►
Page 13
Predictive Modeling
Transaction Analytics
Behavior Modeling
IIA / ISACA Orange County Chapters
Forensic analytics maturity model
Beyond traditional “rules-based queries” – consider all four quadrants
Unstructured
Data
Structured
Data
Low
High
Matching, Grouping, Ordering,
Joining, Filtering
Anomaly Detection, Clustering
Risk Ranking
“Traditional” rules-Based Queries & Analytics
Statistical-Based Analysis
Data visualization, Drill-down
into data, Text Mining
Keyword Search
Traditional Keyword Searching
High
Page 14
Detection Rate
Data Visualization & Text Mining
False Positive Rate
IIA / ISACA Orange County Chapters
Low
Finding hidden money…
Duplicative / split payment analysis
Different
Vendor ID
Similar names
Page 15
Same
Date
Exact
Same
Amount
Different
Invoice #
Some with same
address
IIA / ISACA Orange County Chapters
Same Reference /
Job Code
Vendor / employee conflicts of interest
Vendor Master and Employee Master should not overlap.
Analysis of phone numbers and fuzzy address matches.
Page 16
IIA / ISACA Orange County Chapters
Transaction Risk Scoring
Review breaches on
targeted analytics
Page 17
IIA / ISACA Orange County Chapters
Filter by selected
analytics
Apply various “weightings” to each test
Based on case specific relevance (e.g., scale of 1 to 3 importance)
Adjust weightings for
analytics per
relevance criterion
Page 18
IIA / ISACA Orange County Chapters
Who calls it “bribe expense?”
Page 19
IIA / ISACA Orange County Chapters
Keyword Search Summary
Analyze Keyword Hits by Term, Custodian and Date
Analyze effectiveness of keywords. Understand the effect of keyword hits by
custodian and timeframe to prioritize review and analyze keyword hits.
Page 20
IIA / ISACA Orange County Chapters
Text Mining: “Disbursements Analysis”
Page 21
IIA / ISACA Orange County Chapters
Anti-Bribery & Corruption Analytics
Who said what, where and how much?
Page 22
IIA / ISACA Orange County Chapters
Travel & entertainment
“Who entertained whom, where, what for and for how much?”
Page 23
IIA / ISACA Orange County Chapters
Communication Analytics
“Who emailed whom, when, what and why?”
WHO
`
WHAT
WHEN
WHY
SocialN etw orking
Concept Clustering
Com m unication O ver Tim e
Sentim ent A nalysis
W ho is talking to w hom ?
about what?
over which time period?
how do they feel?
• People-to-people analysis
• Top words mentioned
• When communications occur
• Positive vs. Negative Sentiment
• Entity-to-entity analysis
• Key concepts / topics
• Top 10 negative journal entries
• Map communication lines
to organization chart
• Top or unusual dollar amounts
• Communication spikes
around key business events
• Sensitive words / phrases
• Top 10 angry emails
• Top 10 most concerned emails
• Customer survey analysis
• Employee survey analysis
Page 24
IIA / ISACA Orange County Chapters
New Research: Fraud Triangle Analytics
Page 25
IIA / ISACA Orange County Chapters
The Fraud Triangle*
Applying theory to electronic communications
* Donald R. Cressey's “Fraud Triangle” ; Incentive/Pressure, Opportunity and Rationalization are present when fraud exists. 1.nald R. Cressey's “Fraud Triangle”
; Incentive/Pressure, Opportunity and Rationalization are present when fraud exists.
Page 26
IIA / ISACA Orange County Chapters
EY / ACFE library of ‘keywords’
(Over 3,000 terms in a over a dozen languages so far…)
Rationalization
Incentive/ Pressure
Opportunity
…I deserve it
…make the number
…special fees
…nobody will find out
…don’t let the auditor find out
…client side storage
…gray area
…don’t leave a trail
…off the books
…they owe it to me
…not comfortable
…cash advance
…everybody does it
…why are we doing this
…side commission
…fix it later
…pull out all the stops
…backdate
…the company can afford it
…do not volunteer information
…no inspection
…not hurting anyone
…want no part of this
…no receipt
…won’t miss it
…only a timing difference
…smooth earnings
…don’t get paid enough
…not ethical
…pull earnings forward
Page 27
IIA / ISACA Orange County Chapters
Fraud Triangle analytics – Calculation
Joint EY and ACFE Research Project
Page 28
IIA / ISACA Orange County Chapters
Fraud Triangle Analytics – Research
Bribery Case
Keyword hits as a percentage of total emails
Incentive/Pressure Terms
Opportunity Terms
Rationalization Terms
Investigation timeframe, September 2006 to March 2007
Page 29
IIA / ISACA Orange County Chapters
Interactive dashboard
Fraud Triangle Analytics – Interactive Dashboard
Page 30
IIA / ISACA Orange County Chapters
Electronic Discovery
“eDiscovery”
Page 31
IIA / ISACA Orange County Chapters
Electronic Discovery
“eDiscovery” - FACTS
►Fact: Average number of connected devices per
knowledge worker is expected to reach 3.3 by 2014. This
is up from an average of 2.8 in 2012
►Fact: The deadly cost of ignoring big data: $71.2 Million
in lost revenue per year
►Fact: Through 2013, more than 60% of enterprises will
have some form of Cloud adoption, and the majority will
be exploring Cloud techniques
►Fact: Total eDiscovery costs per gigabyte reviewed are
generally around $18,000
Page 32
IIA / ISACA Orange County Chapters
Top Ten Reasons You Need to Know More
About eDiscovery
10: Your organization has an eDiscovery project budgeted for 2013/2014.
9: You are tasked with developing a business case to justify an eDiscovery project for
2013/2014.
8: Your organization does not have an established eDiscovery Team.
7: Your organization does not have email archiving or is looking to buy a system.
6: Your organization is currently going through a Merger/Acquisition/Divesture.
5: Your organization is saving all their backup and recovery tapes forever to address
litigation issues
4: Your organization has recently issued a legal hold, but is not adequately enforcing
the hold .
3: Your organization has in-house eDiscovery related software, but has technology
gaps or uses outside 3rd-party vendors and/or manual data collection methods
2: Your organization has more than 10 major cases per year, and/or has a wide
dispersed network with thousands of potential custodians.
1: Your organization has made the front page of the Wall Street Journal and not
in a good light.
Page 33
IIA / ISACA Orange County Chapters
Why Do Corporations Care About eDiscovery?
Why does the Legal department care? On average, “American businesses are
spending between $2.5 and $4.0 million
per year for e-Discovery, per billion
-Wants to defend the corporation
dollars in sales… making e-discovery the
-Avoiding Sanctions
largest controlled cost in American
-Mitigating Risk
business.1”
-Minimizing Costs
-CFO pressure
-Wants fewer lawyers doing review saving time and money
Why does the IT department care?
1 Source:
Cohasset Associates. “The Eternal Charter: Improving
Corporate Governance through Compliance and Assured Records
Management.” June 2005.
-It’s their network, hands off!
-Time
-Resources
-They will end up doing the work operationally
Page 34
IIA / ISACA Orange County Chapters
eDiscovery - What Every IT/Audit/Compliance/Legal
Professional Should Know
►
►
Why do the Courts care about Electronically Stored Information (ESI)?
More than 95 percent of all information is now electronic:
•
•
•
•
ESI is normally stored in much greater volume than are hard copy documents.
ESI is dynamic, in many cases modified simply by turning a computer on and off.
ESI can be incomprehensible when separated from the systems that created it.
ESI contains non-apparent information, or metadata, that describes the context of
the information and provides other useful and important information.
►
IT professionals need to understand the Rules of Evidence and Rules of Civil
Procedure as much as they do the server technology and storage area
networks
►
Lawyers, on the other hand, need to understand how IT professionals
manage their business and at a high level, what is on the network and where.
►
The confluence of IT and Law has subsumed litigation in the U.S.
Page 35
IIA / ISACA Orange County Chapters
ESI Sources – Going Beyond Email
80%
Email (and attachments)
60%
General office productivity documents
49%
Database records
41%
Invoices and other customer records
36%
Financial statements
Laptops
29%
Phone call recordings & other audio files
Desktops
25%
Digital images
File Server
21%
Instant messages
Email Server
16%
Video files
5%
Other
0%
10% 20% 30% 40% 50% 60% 70% 80% 90%
IIA / ISACA
Orange
County
Chapters
Page 36Source: ESG Research Report, Electronic Discovery Requirements
Escalate,
November,
2007
How much potentially relevant data do you
have on your network?
►
►
►
►
►
CD = 650 MB = 50,000 pages. DVD = 4.7 GB = 350,000 pages. DLT Tape = 40/80 GB = 3 to 6 Million pages.
Super DLT Tape = 60/120 GB = 4 to 9 Million pages.
***************************
Page Estimates:
1 MB is about 75 pages;
1 GB is about 75,000 pages (pick-up truck full of documents).
Aver. pgs. per email: 1.5 (100,099 pages per GB).
Aver. pgs. per word document: 8 (64,782 pages per GB).
Aver. pgs. per spreadsheet: 50 (165,791 pages per GB).
Aver. pgs. per power point: 14 (17,552 pages per GB).
***************************
For the average .PST or .NSF Email File:
100 MB .PST file is 900 emails and 300 attachments.
400 MB .PST file is 3,500 emails and 1,200 attachments.
600 MB .PST file is 5,500 emails and 1,600 attachments.
A 1.00 GB .NSF file is 9,000 emails and 3,000 attachments.
A 1.5 GB .NSF file is 13,500 emails and 4,500 attachments.
***************************
Note: Many variables will affect ALL of the actual numbers above, including especially large image and video files, and recursive
files.
►
***************************
Bits and Bytes Sizes:
•8 bits are equal to 1 byte (one or two words),
•1,024 bytes are equal to 1 kilobyte (KB).
•1,024 kilobytes (KB) are equal to 1 megabyte (MB or Meg).
•1,024 megabytes are equal to 1 gigabyte (GB or Gig) (truck full of paper).
•1,024 gigabytes are equal to 1 terabyte (TB) (50,000 trees of paper).
*ACFE 2008
the
Nation
On Occupational
Fraud
•1,024Report
terabytestoare
equal
to 1 petabyte
(PB) (250 Billion
Pgs. of Text).
•1,024 petabytes are equal to 1 exabytes (EB) (1 000 000 000 000 000 000 bytes).
Page 37
IIA / ISACA Orange County Chapters
New Court Rules for eDiscovery
►
Federal Rules of
Civil Procedure
►
►
►
►
Went into effect:
December 1, 2006
►
Requires meetings to discuss
e-discovery
Requires production of all potentially
relevant reasonably accessible data
Defines "electronically stored information"
Deals with inadvertent waivers
of privilege
Enables requesting party, in first instance,
to specify desired production formats
Sets up possible "safe harbor"
for lost data
*ACFE 2008 Report to the Nation On Occupational Fraud
Page 38
IIA / ISACA Orange County Chapters
What are the eDiscovery requirements
from the FRCP?
►
Rule 16 – Court orders regarding electronic discovery much earlier
►
Rule 26 – Early discussions and disclosures of electronic information.
Pre-meeting must occur between teams at companies involved in a lawsuit.
Companies must represent where and how data is stored. Technology must
be in place to provide access to requested information
►
Rule 33 – Responding to interrogatories by providing access to systems
►
Rule 34 – Electronically stored information (ESI) gets its own category.
Organizations must deliver files & emails in the format requestors define.
Formats are usually native to retain hidden metadata (which must be
extracted). This rule precludes most search solutions that convert or do not
extract metadata
►
Rule 37 – Sanctions and good faith. Codifies legal hold standards to
ensure automated systems don’t delete evidence. Companies cannot delete
files or emails involved in an ongoing lawsuit. Precludes search solutions that
don’t include data classification or management
Page 39
IIA / ISACA Orange County Chapters
What are the eDiscovery requirements from the FRCP?
►
eDiscovery
►
►
►
FRCP Rule change — Specifically address eDiscovery; add references to
electronically stored information (ESI); codified principles set forth in Zubulake
►
►
►
Electronic Discovery - is the compiling, storing and securing of digital electronically stored
information, such as e-mail, files and other data.
Electronically Stored Information (ESI) – is a digital file, document, email or record that was
originally created on a computer or a paper document that has been scanned.
Defensible Process. The amendments underscore the importance of a systemized
eDiscovery processes. Organizations need to establish a repeatable and defensible
eDiscovery process.
The Duty to Preserve ESI Applies Only to Relevant Data. The Advisory Committee Notes
provide that preservation efforts need only be “reasonable” and “narrowly tailored” to relevant
information.
Early Attention Requirement. Rules 16(b) and 26(f) require a pre-trial conference,
“Meet and Confer”, about eDiscovery.
► Rule 34(b) Production Requirement. Rule 34(b) permits the requesting party
*ACFE
2008 Report
to the
On Occupational
Fraud
specify
the form
ofNation
production;
however,
absent specification, “must produce
information in a form in which it is ordinarily maintained.”
Page 40
IIA / ISACA Orange County Chapters
to
eDiscovery - What Every IT/Audit/Compliance/Legal
Professional Should Know
►
Generally after a lawsuit and before the trial there is a period of time referred
to as Discovery where each party asks questions about the claims and
defenses to avoid surprises at trial.
►
Evidence collected during Discovery is used to prove or disprove claims.
►
If a party withholds information (evidence), it may be penalized by
sanctions, adverse inferences, and losing the trial
►
There are four primary categories of Discovery:
1.
2.
3.
4.
Page 41
Written questions referred to as interrogatories;
Requests for the production of documents and things;
Requests for admissions; and
Oral testimony called depositions.
IIA / ISACA Orange County Chapters
eDiscovery - What Every IT/Audit/Compliance/Legal
Professional Should Know
►
Interrogatories are questions: ex., "Describe the procedures for daily backup
of the e-mail system” , “List all employees who had passwords to the Oracle
Financial database between Jan. 20, 2003 and May 9, 2005”
►
Document: ex., “All .doc, .xls and .ppt with any of these 50 keywords”; “All emails from certain individuals during a defined time period”, “all reports from
a certain financial system”, or “copies of correspondence between specific
individuals or departments”.
•
►
For purposes of this notice, “Electronically Stored Information” shall include, but not be limited to, all text files (including word processing
documents), spread sheets, email files and information concerning email (including logs of email history and usage, header information
and “deleted” files), Internet history files and preferences, graphical image files (including “.JPG, .GIF, .BMP and TIFF” files), data bases,
calendar and scheduling information, computer system activity logs, and all file fragments and backup files containing ESI.
Request for admissions are a series of questions to admit true or deny. For
instance, "Admit or deny that all e-mails sent by Jane Doe were not saved
between June 27, 2001 and Sept. 10, 2003”
Depositions Under the discovery rules, IT professional can be deposed by a
category of "the person or persons most knowledgeable" about lists of things
like e-mail, backups, databases, Web sites related to the lawsuit, and the
*ACFE 2008 Report to the Nation On Occupational Fraud
like.
►
Page 42
IIA / ISACA Orange County Chapters
eDiscovery for Compliance Enforcement/Audit
►
eRecords Management Auditing - Conduct proactive records
management audits to identify all records that are in violation of your
policy, if documents match the search criteria, they should be
preserved as evidence for later use.
►
Need the ability to search, collect any file types or combination of
search terms for collection of electronic evidence
►
Need the ability to reveal confidential documents anywhere on
your network and collect it for evidentiary purposes.
►
Protect your company's Intellectual Property and assist with
maintaining accurate financials
Page 43
IIA / ISACA Orange County Chapters
Electronic Discovery Reference Model
www.edrm.net
*ACFE 2008Information
Report to the Nation
On Occupational Fraud
Technology
Page 44
IIA / ISACA Orange County Chapters
Legal
Enterprise-wide eDiscovery Technology
Reduce Cost and Risk across the EDRM
Electronic Discovery Reference Model
(www.edrm.net)
Current
Scope
Processing
Preservation
Information
Management
Identification
Review
Production
Presentation
Collection
Analysis
VOLUME
Identification – Data Mapping,
Early Case Assessment
• Metadata Scan
• Custodian, File Type,
Date Range, Keyword
• Volume of Expected ESI
Collection
• Location for Data
Mapping
• Sampling
• Keyword
Page 45 Testing
RELEVANCE
Search, Collection & Forensic
Preservation
Processing
•
•
•
•
•
•
•
•
By Custodian / SID
By Operator
By File Type
By Keyword, Boolean, Proximity,
Date Range, Pattern, Hash Value
• By Metadata
• By Location, range of IP
• In any language
Deduplication
Privilege Forking
Review Grouping
Further Culling &
Filtering when
scope is narrowed
• Chain of Custody
and Authenticity
IIA / ISACA Orange County•Chapters
Load Files or Native
Review & Analysis
• Responsiveness
• Privilege
Determination
• Confidential
• Hot Docs
• Concept Grouping
• Near-duplicates
• Quality Control
The eDiscovery Funnel
Page 46
IIA / ISACA Orange County Chapters
Solidifying Your eDiscovery Readiness
Your eDiscovery Program must:
Your eDiscovery Team must:
•
Mitigate Risks
•
Communicate
•
Reduce Costs
•
Delegate
•
and help you Gain Control over
eDiscovery and ESI.
•
and Maintain Control over
eDiscovery and ESI.
Primary Technology Solutions
• eDiscovery
• Data Classification
• Incident Response
• Forensics
• Archiving/Storage
Page 47
Other Related Technologies
• Policy & Configuration Management
• Data Loss Prevention
• Encryption and Key Management
• eMail Security
IIA / ISACA Orange County Chapters
Establish an eDiscovery/Compliance Team
►
“Successful CIOs should enhance their relationships with
internal legal and corporate-affairs teams and be prepared
to engage productively with regulators. They will need to
seek solutions that meet government mandates at
manageable cost and with minimal disruption.”
- McKinsey Quarterly, February 2009
►
Who better than the IT folks to know where ESI is?
The team should be composed of both Legal and IT
Large and small companies alike are assembling teams to lower cost
and risk
In order to make sure that everyone’s on the same page, need a
dedicated Legal person to deal with IT, and a dedicated IT person to
*ACFE 2008 Report to the Nation On Occupational Fraud
deal
with Legal (or a liaison)
►
►
►
Page 48
IIA / ISACA Orange County Chapters
Establish an eDiscovery/Compliance Team
Drivers for Working Together
► eDiscovery and civil litigation
►
►
►
►
Compliance and regulatory issues
►
►
►
►
►
►
FRCP
Recent Case law
International trends
Sarbanes-Oxley whistleblower hotlines, accounting systems, etc.
HIPAA and policing health information
PCI requirements need to be met for credit card numbers and information
SEC and banking-regulated companies and all that they need to deal with
FDA requires many pharmaceutical systems be “validated”
Internal investigations
Fraud
► Policy violations
► Firing
► Internal audit
*ACFE 2008 Report to the Nation On Occupational Fraud
►
Page 49
IIA / ISACA Orange County Chapters
Translate to each others language
Legal Help IT Understand eDiscovery
► In-house legal teams should meet with IT (if they aren’t already) to help them
better understand the nature of eDiscovery, particularly the “upstream” parts
of the process (specifically, identification, preservation, and collection) which
IT tends to be responsible for
► With an understanding of the nature of eDiscovery, IT can improve its ability
find the right documents, avoiding over-collection and reducing
“downstream” processing costs.
► In addition, new eDiscovery technologies are making it increasingly easy for
legal to own more of the process, reducing the eDiscovery burden on IT
IT Help Legal Understand IT Reality
► Conversely, IT should provide advice and mentoring as legal seeks to bring
eDiscovery platforms in-house
► For legal teams, bringing eDiscovery in-house may seem daunting, but
enterprise software has been around for a long time, and learning from IT’s
experiences can make the process far less intimidating
► What is an achievable deadline for collection and production?
Page 50
IIA / ISACA Orange County Chapters
The Committee of Decision-Makers
Corporate Legal Department
Corporate Litigation Support Department
Internal Audit Department
Information Risk Department
Compliance Department
IT Department
► CIO
► CISO (may run the project from IT side)
► Corporate Security Manager
► Networks
► Desktops
► Email
► Servers
► Archive
► Back-Up
Page 51
IIA / ISACA Orange County Chapters
Establish an eDiscovery/Compliance Team
eDiscovery/Compliance Teams In Action
► Focus on searching for and collecting ESI
► Enforce records management and compliance policies
► Police the data that’s out there – determining when it
should be deleted or when it should be moved to storage,
etc.
► Help companies minimize cost and also minimize risk
► Have increased consistency in terms of how they search
because they’re an internal team that does it all the time
► As such, they improve the quality of that search and
collection, as well as increasing its responsiveness
Page 52
IIA / ISACA Orange County Chapters
Establish an eDiscovery/Compliance Team
eDiscovery/Compliance Team Responsibilities
►
Orchestrating the relationship between IT and Legal
►
Overseeing eDiscovery, Compliance, Risk, Audit, & Cyber Security
►
Ensuring third-party vendor compliance
►
Developing strategies and tactics to manage risk
►
Establishing privacy policies to advise organization on how data will be
protected, identified, preserved, collected, processed and reviewed
►
Creating a response plan in the event of a security breach
*ACFE 2008 Report to the Nation On Occupational Fraud
Page 53
IIA / ISACA Orange County Chapters
Establish an eDiscovery/Compliance Team
eDiscovery/Compliance Teams are being created across
the board
►
There’s a perception that smaller companies aren’t doing
this, but that’s not the case.
►
A lot of small companies are heavily regulated, and if
they’re outsourcing e-discovery, that can be very, very
expensive.
►
Many companies are seeing huge savings by bringing
this in house and controlling it.
Page 54
IIA / ISACA Orange County Chapters
eDiscovery/Compliance Team:
Summary
►
►
►
►
►
Don't wait for a security issue to introduce your IT and your legal
departments. Be highly proactive.
Use eDiscovery to get a jump on information security issues.
Consider hiring an Electronically Stored Information (ESI)
Coordinator to help you bridge IT and Legal, as now recommended
by a number of judicial districts.
Consider enlisting outside, objective experts to handle tasks such as
conducting a security assessment, preparing a crisis communications
plan and reviewing the customer notification requirements. Their
input will help you respond quickly to any breaches, and will help
prove you did your best to provide reasonable and prudent
safeguards.
Don't get caught up in security theater, i.e., countermeasures that
provide the feeling of security while doing little or nothing to actually
improve it. For example, what we are doing today is good enough…
(it could be better!)
Page 55
IIA / ISACA Orange County Chapters
Legal Hold Business Challenges
► “The basic principle that an organization has a duty to preserve relevant
information in anticipation of litigation is easy to articulate. However, the precise
application of that duty can be elusive.”
►The Sedona Conference® Commentary on Legal Holds: The Trigger & The
Process, August 2007
► Many current litigation hold solutions do not provide an integrated means to
systematically collect & process data from custodians subject to litigation holds
► Ensuring that the legal hold is enforced
► Keeping custodians out of the mix
► “Collect to Preserve” vs “Preserve in Place”
Page 56
IIA / ISACA Orange County Chapters
Duty to Preserve - Legal Hold Notification
►Duty to Preserve
► Begins when counsel for a party reasonably anticipates litigation, i.e., knows or
should have known, that evidence is relevant to existing or future litigation
►Legal Hold Notice
► Is a communication issued as a result of current or anticipated litigation, audit,
investigation or other such matter
►Suspending the normal disposition or processing of records
►Legal holds can encompass procedures affecting active data and backup tapes
►The duty to preserve ESI BEGINS when you reasonably anticipate litigation,
for EVERY case.
► First decision – Is this a case, yet?
► Next - Scope and Method
► WHAT needs to be preserved?
► HOW should it be preserved?
► Key Consideration - Accessibility
► Difference between what needs to be PRESERVED and what needs to be
REVIEWED AND PRODUCED
► Tiered approach
Page 57
IIA / ISACA Orange County Chapters
Records Retention and the Legal Hold
►
Records Retention Policies Because ESI has become fundamental
to litigation, organizations need to have a records retention policy;
otherwise, how can an organization without such a policy explain to a
judge why certain ESI was retained and other deleted?
►
Hit the PAUSE button on Records Retention Policy for a Legal Hold
►
When you become aware that a lawsuit may occur, the records
retention policy must be changed regarding relevant records to
the litigation. The legal term for this is "litigation hold;" however, the
rules of evidence have always required potential litigants to save
evidence, and if they destroy critical evidence, they lose their case
because of the intentional destruction which is call spoliation.
Page 58
IIA / ISACA Orange County Chapters
How to Select and Prepare the FRCP 30(b)(6)
Deposition Witness in eDiscovery
►
►
►
►
His or her role is to testify not on the facts of the case, but on a company’s
operations, such as its IT infrastructure or accounting practices.
Frequently referred to as the “Voice of the Corporation”, the 30(b)(6) witness’s
testimony represents the knowledge of the entity, not of the person being deposed.
In the context of eDiscovery, this witness is often called to testify on the steps the
corporation took to find and produce responsive documents to ensure discovery
was diligently completed in good faith.
Topics can vary greatly
►
►
►
►
►
►
►
►
►
►
►
Page 59
Qualifications and organizational structure
Information systems
Software and email
Records management
Alternative sources of electronic information
Legacy systems
Backup and restoration procedures
Production of ESI in other lawsuits
Location and access to ESI
How ESI is maintained
How this data was preserved for the subject litigation
IIA / ISACA Orange County Chapters
The Rule 26(f) Meet & Confer
► FRCP §26(f) - A pre-trial conference provision mandates that the parties
shall confer with regard to anticipated electronic discovery issues, and address
these issues to court at the preliminary conference.
►
►
►
►
►
►
►
►
►
Page 60
Implementation of a data preservation plan
Identification of relevant data
Scope, extent and form of production
Anticipated cost of data recovery and proposed
►initial allocation of such cost
Disclosure of the programs and manner in which the data is maintained
Identification of computer system(s) utilized
Identification of the individual(s) responsible for data preservation
Confidentiality and privilege issues
Designation of experts
IIA / ISACA Orange County Chapters
Timeline-Initiation to Scheduling Order
Initiation of Litigation
- Day 1 -
Scheduling Order
- Day 120 -
•Deadline for confer
– Disclosure or discovery of ESI should be
handled…
– Brief description of parties’ proposals (Form 35)
•Begin to discuss and document proposals
–
Secure a cost effective reasonable agreement favorable to your
case
–
Create a record of good faith and cooperation
–
Maximize cost shifting opportunities
–
Solidify preservation, privilege, reduction strategies
•Assess electronic evidence early
– Collect and review subset of key custodians/tapes
– Extrapolate for budget and scheduling discussions (total cost of
discovery)
– Test search term, sampling and other culling strategies
– Test privilege/clawback/quick peek strategy
•Meet with key custodians, IT, Records management
–
Determine compliance with legal hold
–
Ascertain costs of preservation
–
Assign tasks for mandatory disclosures (gather existing
documentation)
–
Assess appropriate “person most knowledgeable”
•Complaint served
–
Latest day for preservation obligation to attach
–
Litigation hold notices
–
Suspension of appropriate destruction policies
Page 61
IIA / ISACA Orange County Chapters
Difference between eDiscovery & Email Archiving
Solutions
►For eDiscovery, relevant information may be found in:
►unstructured,
►semi-structured or
►structured data sources
►dispersed across networks
►on desktops, laptops, servers, shares, removable storage media, etc.
►eDiscovery is the practice of identifying, collecting, preserving, processing,
reviewing and producing relevant ESI from all data sources
►Email archiving systems, on the other hand, were designed to conduct email
management and work with the set of emails that reside in the archive, and do
not extend to all data on the network.
►Even if an organization uses an archive system, they must have a scalable
capability to identify potentially relevant information from the
unstructured/ unmanaged environment.
Page 62
IIA / ISACA Orange County Chapters
Difference between Archiving vs Backup
Solutions
Archive
►
►
Backup
Archives are the primary sole copy of static or
persistent information
Value is retained for future reference (months,
years or decades)
►
Authenticity and inalterability must be assured
►
It is typically in its final form, and subject to
limited or no modification
►
Archives focus on access and retrieval of a
specific informational items vs. an entire
volume restore
►
Archival processes often include specific
timeframes; including restore or deletion
►
Archives may be refreshed regularly and the
information stored is maintained long-term
Page 63
►
Backups constitute raw content and
contain no means to enable search.
►
Data integrity is not guaranteed. Due to
age, mishandling or corruption (tapes)
►
Backups are a snapshot of data that is
generated at that point in time.
Between backups, data could have
been created, modified and/or deleted
Backups are secondary copies of
primary information.
►
►
Backups provide short-term protection
of production data to ensure business
continuity, and are systematically
overwritten.
►
Backup solutions are appropriate
solutions for business continuity and
disaster recovery.
IIA / ISACA Orange County Chapters
Market Needs: Handling ESI
► “The
real opportunity for worthwhile ROI lies in reducing the
number of responsive documents to a sufficient set. This is the
heart of Early Case Assessment…
►
The new tools will enable attorneys to handle multiple matters with
overlapping custodians more efficiently and to take a consistent approach
to the handling of ESI from case to case.”
–
Page 64
Gartner report, "Reduce the Cost and Risk of E-Discovery in 2009” (ID Number:
G00164554, 9 January 2009)
IIA / ISACA Orange County Chapters
eDiscovery / Legal Hold Case Law:
Notification alone is insufficient
►
In re Hawaiian Airlines, Inc.
The Court issued an adverse inference instruction
because they “simply told [the custodian] to
preserve all evidence and trusted him to comply”,
rather than taking reasonable steps to prevent
spoliation, the Company facilitated misconduct
even though it was the custodian who acted in
bad faith. 2007 WL 3172642 (Bkrtcy. D.Hawaii
October 30, 2007).
►
In re NTL, Inc. Securities Litigation
“Although NTL sent out hold memos in March
and June 2002… those hold memos were not
sufficient…” (citing Zubulake). Failure to
implement proper ESI preservation process is
“gross negligence” 244 F.R.D. 179 (S.D.N.Y.
2007)
Page 65
IIA / ISACA Orange County Chapters
eDiscovery / Legal Hold Case Law:
Custodian Self-Collection is insufficient
►
Wachtel v. HealthNet, 239 F.R.D. 81 (D.N.J. 2006); Court states
that “Health Net’s process for responding to discovery requests was
utterly inadequate”
► “Health Net relied on the custodians within the company to search
and turn over whatever documents they thought were responsive,
without verifying that the searches were sufficient. The process, in
sum, was one of looking for selected specific documents by a specific
person rather than all responsive documents from all Health Net
employees who had such documents. Many of these specific
employee-conducted searches managed to exclude inculpatory
documents that were highly germane to Plaintiffs' requests.”
►
Cache La Poudre Feeds, LLC v. Land O’Lakes, Inc., 244 F.R.D.
614 (D.Colo. 2007); Court faults Land O’Lakes for simply directing
employees to produce relevant information, and then relied upon those
same employees to exercise their discretion to determine what
information to save
Page 66
IIA / ISACA Orange County Chapters
No Requirement to Preserve
Non-Relevant Data for Litigation
►
The Duty to Preserve Extends to Only Potentially
Relevant Information
►
Zubulake v. UBS Warburg LLC, 220 F.R.D. 212, 217
(S.D.N.Y. 2004) (“Zubulake IV”)
►
►
“Clearly” no duty to “preserve every shred of paper, every e-mail
or electronic document, and every backup tape…Such a rule
would cripple large corporations.”
However, Previously No Effective Way to Separate the
Wheat from the Chaff at the Point of Collection
►
Page 67
Result: Either non-compliance or over-collection
IIA / ISACA Orange County Chapters
How broad reaching is the hold
on potentially relevant documents?
►
Micron Technology, Inc. v. Rambus, Inc., C.A. No. 00-792-SLR on
January 9, 2009, declaring certain patents unenforceable as a
sanction for spoliation.
►
In a suit for patent infringement, Micron claimed Rambus employed
a document retention policy that destroyed documents while
they had a duty to preserve.
►
The Court said that Rambus was an “aggressive competitor” so
should have foreseen litigation as far back as December
1998. All relevant documents destroyed by Rambus after that time
was spoliation.
►
As a sanction, the Court decided the patents at issue were not
enforceable against Micron.
Page 68
IIA / ISACA Orange County Chapters
eDiscovery changes the way we think
about data storage and retention policies
►
Omnicare, Inc. v. Mariner Health Care Mgmt. Co., 2009 WL 1515609
►
In Omnicare, the Court ruled that just because data is on a backup tape
doesn’t automatically make it ‘not reasonably accessible.’
►
Omnicare sued Mariner for breach of contract and moved to compel Mariner
to restore backup tapes to retrieve old emails deleted pursuant to their
data retention policy.
►
The Court looked to Zubulake to analyze the cost-shifting argument, and
decided that cost-shifting was not warranted in this case, noting that just
because “ESI is now contained on Backup Tapes instead of in active
stores does not necessarily render it not reasonably accessible.”
►
Nonetheless, the Court opted not to order the restoration, opting instead for
the active file sampling Mariner proposed.
Page 69
IIA / ISACA Orange County Chapters
Zubulake Court’s suggestion of a Defensible
collection and preservation Process
1.
Run system-wide keyword search
2.
Use a broad list of search terms (from Legal)
3.
Search for a limited time-frame (determined by Legal)
4.
Segregate and preserve responsive documents (the “hits”)
5.
Opposing counsel will make a demand for production of documents
6.
Legal negotiates with opposing counsel on proper list of keywords
7.
This keyword search conducted only against the preserved “hits”
Page 70
IIA / ISACA Orange County Chapters
If you cannot demonstrate a Defensible Process
►
Mudron v. Brown & Brown, Inc., 2005 WL 645927
(N.D.Ill., Mar. 17, 2005)
Page 71
►
Defendant apparently had not conducted its own comprehensive search and
production of digital data
►
Plaintiff “filed a motion for discovery sanctions and other relief alleging that he has
been consistently denied electronic data that is in [defendant's] control which may
be relevant . . . [The Court] ordered that [plaintiff’s] forensic expert be allowed
access to [defendant’s] computer drives to obtain forensic images.”
►
Court granted plaintiff’s expert access to company’s network
►
compromised legal position
►
substantial disruption of operations
IIA / ISACA Orange County Chapters
Four Steps for Legally Defensible Retention
Policies
►
Step 1: Companies must have a clear written document retention policy and
schedule that meets its business needs and is fully endorsed by senior
management. The policy should define when, where and by whom those
records that are required by law or contract, or are otherwise deemed valuable to
the company, are routed to appropriate archives, and those records no longer
required are to be properly destroyed. Companies also must specify the
means of destruction.
►
Step 2: Companies must take reasonable steps to ensure that this policy is
effectively communicated to employees, and is actually followed (e.g.,
periodic audits, compliance days, certifications). Haphazard implementation
will not provide a defence in a negligent spoliation claim.
►
Step 3: Companies must enact administrative procedures that will
immediately stop the routine destruction of records when and if they become
the subject of corporate governance, regulatory or legal concerns.
►
Step 4: Companies must guide and train all employees on how to prepare
effective, accurate records.
Page 72
IIA / ISACA Orange County Chapters
eDiscovery: Sanctions Result In 35% of Decisions
Addressing These Issues
►
Mr. Perry L. Segal, an IT executive turned e-discovery attorney and consultant reports in “EDiscovery: New Rules, Big Headaches,” published in the September 2009 California Lawyer.
►
►
“… it’s estimated that out of all case law that
addresses e-discovery issues, more than 35%
result in sanctions.”
Tackle e-discovery with the “W5+H” approach:
► Who should be involved?
► What data are you looking for?
► When are the due dates?
► Where is the data?
► Why is it relevant?
► And how will you comply?
Page 73
IIA / ISACA Orange County Chapters
Selected Sanctions Decisions:
►
Coleman Holdings v. Morgan Stanley, (Fla. App. 4th Dist. Mar. 21,
2007) – $1.45 billion in compensatory and punitive damages.
Judgment reversed, but …
►
Qualcomm, Inc. v. Broadcom Corp., 2007 WL 2900537 (S.D.Cal
Jan., 2008) - court sanctioned Qualcomm for concealing thousands
of pages of relevant e-mails $8.5 million atty fees.
►
In re Sept. 11th Liab. Ins. Coverage Cases, 2007 WL 1739666
(S.D.N.Y. June 18, 2007) - Court Imposes $1,250,000
►
In re Seroquel Prod. Liab. Litig., 2007 WL 2412946 (M.D.Fla. Aug.
21, 2007) - Sanctions for Purposeful Sluggishness in Discovery
Page 74
IIA / ISACA Orange County Chapters
Court Sanctions Qualcomm
$8,568,633, Orders Counsel Sanctioned
►
Qualcomm, Inc. v. Broadcom Corp., 2007 WL 2900537 (S.D.Cal
Sept. 28, 2007)
►
►
►
►
Page 75
Cross-examination of the plaintiff’s witness revealed the existence of relevant emails that the court later held were “the tip of the iceberg” in an attempt to
conceal over 200,000 pages of relevant e-mails.
The court found by clear and convincing evidence that Qualcomm’s counsel
engaged in misconduct by providing calculatedly misleading and false discovery
responses, asserting patently false statements of fact during motion hearings,
minimizing the significance of missing e-mail at trial, continuing through post-trial
activity.
The judge characterized the discovery abuses as, “an organized program of
litigation misconduct” and ordered the plaintiff’s attorneys to demonstrate why they
should not be sanctioned, without use of documents protected by the attorneyclient privilege.
See also Qualcomm, Inc. v. Broadcom Corp., 2007 WL 2261799
(S.D.Cal. Aug. 6, 2007); Qualcomm, Inc. v. Broadcom Corp., 2007 WL
2296441 (S.D.Cal. Aug. 6, 2007); Qualcomm, Inc. v. Broadcom Corp.,
2007 WL 1031373 (S.D.Cal. March 21, 2007).
IIA / ISACA Orange County Chapters
Court Imposes $1,250,000 in Sanctions
for eDiscovery Violations
►
In re Sept. 11th Liab. Ins. Coverage Cases, 2007 WL
1739666 (S.D.N.Y. June 18, 2007)
►
►
►
Page 76
Port Authority and Westfield moved for sanctions, alleging Zurich’s
position throughout the pleadings was objectively unreasonable in
violation of Rule 11 of the Federal Rules of Civil Procedure (“FRCP”) and
the discovery abuses violated FRCP 37.
The court held Zurich and its counsel liable for $1,250,000 based on
violation of both Rules.
As Zurich deleted the electronic version of an essential document
and possessed the paper version for over three years before
producing it, the lease holders were successful in meeting the
burden for the court to impose sanctions. The court determined the
$1,250,000 was sufficient to deter repetition of such conduct or
comparable conduct by others similarly situated.
IIA / ISACA Orange County Chapters
Automated Predictive Coding
Validation and defensibility
Silva Moore v. Publicis Groupe, No. 11 Civ. 1279
(ALC) (S.D.N.Y. Feb. 8, 2012)
►Da
►
►
“What the Bar should take away from this Opinion is that
computer-assisted review is an available tool and should
be seriously considered for use in large-data-volume cases
where it may save the producing party (or both parties)
significant amounts of legal fees in document review.”
“As with keywords or any other technological solution to
eDiscovery, counsel must design an appropriate process,
including use of available technology, with appropriate
quality control testing, to review and produce relevant
ESI while adhering to Rule 1 and Rule 26(b)(2)(C)
proportionality. Computer-assisted review now can be
considered judicially-approved for use in appropriate cases.”
Page 77
IIA / ISACA Orange County Chapters
Predictive Coding
Validation and defensibility
►Global
Aerospace, Inc. v. Landow Aviation, L.P.
No. CL 61040 (Vir. Cir. Ct. Apr. 23, 2012)
► Court-ordered
use of technology assisted review over opposing party’s objection
► In response to defendant’s motion requesting that either predictive technology be
allowed or that plaintiff pay any additional costs associated with traditional review
►Kleen
Products v. Packaging Corp. of America,
No. 10-C-5711 (N.D. Ill.)
► Technology
assisted reviewed leveraged to ensure the accuracy of defendants'
document production
► Judicial endorsement of Sedona principles regarding cooperation & quality
►EORHB,
Inc., et al v. HOA Holdings, LLC, C.A.,
No. 7409-VCL (Del. Ch. Oct. 15, 2012)
► Bench
order requiring both parties to use technology assisted review or “to show
cause why this is not a case where [TAR] is the way to go.”
Page 78
IIA / ISACA Orange County Chapters
Document review development
Bankers boxes
► Start/stop sheets
► Photocopies
► Manual logs
► Lawyer driven
►
1990
Megabytes
► Linked databases
► File conversion
► Extracted text
► Outsourcing
►
1995
2000
Computers
► Databases
► Imaging
► Tagging
► OCR
►
2005
2010
Gigabytes
► Native review
► Data processing
► Keywords
► Project managers
►
ESI volume
Page 79
Terabytes
► TAR
► Dual monitors
► Data analytics
► Managed review
►
IIA / ISACA Orange County Chapters
Document review development
Petabytes
► Ernst & Young's
technology-assisted
review
► SME reviewers
► Statistical validation
►
The world's information is more than doubling
every two years, with an estimated 1.8 zettabytes
created in 2011 – the equivalent
of roughly 1.8 billion desktop computers
►
90% of the data in the world today has been
created in the last two years alone
►
The continued application of antiquated workflows
to modern data sets results in expensive reviews
that take months or years to complete with no
measurable degree of quality or consistency
►
Ernst & Young's technology-assisted review
solutions allow you to find what
is relevant much faster – often in only a matter of
weeks – with statistically validated results that are
repeatable, transparent
and defensible
►
2012
Page 80
IIA / ISACA Orange County Chapters
Technology Assisted Review landscape
Standard processing and review
Processing
Pre-culling
Linear Review
►Deleted file recovery
►Aliases
►Review platform
►Deduplication
►Dates
►Workstations
►De-NISTing
►File types
►First level and quality review
►Text extraction
►Locations
►Email threads
►Metadata extraction
►Email Folders
Search
Basic search
Intermediate Search
Advanced Search
Expert Search
►Key terms
►Key terms in context
►Concepts
►Natural Language Processing
►Boolean
►Key term variants
►Synonyms
►Ontologies
►Inter-document context
►Term disambiguation
►Clustering
►Acronyms
Automation
Advanced Search Technology
Page 81
Machine Learning Algorithms
Sampling
►Entity extraction
►Statistical clustering
►Random
►Tools to build complex queries
►Latent Semantic Analysis (LSA)
►Systematic
►Social Networks
►Probabilistic Latent Semantic Analysis (pLSA)
►Stratified
►Language identification
►Vector space model
►Proportional
►Segmentation
►Latent Dirichlet Allocation (LDA)
►Biased
IIA / ISACA Orange County Chapters
Technology-assisted review observations
► Faster
and more efficient first level review
►
Model substantially exceeds first level review performance
►
Demonstrates the ability to adapt and improve
►
Demonstrates the ability to predict high quality results and bulk
code very large datasets in a compressed timeframe
► Deeper
insight into the data
►
Demonstrates the ability to identify key trends within the data
►
Galvanizes review definition by forcing conflict resolution
► Better
advocacy
►
Better understanding of how data relates to themes
►
Identify first order and second order hot documents
Page 82
IIA / ISACA Orange County Chapters
Predictive coding
methodology
Scientific methodology insists that hypotheses be tested in controlled
conditions which can be reproduced by others.
1. State the problem
Question
2. Gather information
3. Form a hypothesis
Research
4. Test the hypothesis
Hypothesis
5. Record and analyze data
Test
6. State the conclusion
Analyze
7. Repeat the work
The requirements of
experimental control and
reproducibility diminish the
effects of cognitive bias
Page 83
True
False
Report
Repeat
IIA / ISACA Orange County Chapters
Predictive coding
decision tree
Create
project
Initial
sampling
Conflict
resolution
Conflict
resolution
Results
satisfactory?
N
Model
building
sampling
N
Results
satisfactory?
Y
Bulk
code
Y
Validation
round
Y = Yes
Page 84
N = No
Conflict
resolution
IIA / ISACA Orange County Chapters
Reporting
Predictive coding
data driven process
Decision points informed by real-time results
Conflict
Resolution
Create Project
Rounds
Any number of predictive
coding projects can be
created and controlled
within a single Relativity
project.
A round of stratified
random sampling is
conducted by a small team
to drive system results.
A project can be used for
assorted purposes,
including identifying high
precision sets for quick
turnaround rolling
productions and
identification of hot
documents.
A team typically consists of
three to four experts.
An initial sample jumpstarts the process of
training the system and
dividing the dataset into
desired section.
Iterative model
improvement samples may
be required for further
system training.
A final sample is drawn to
validate the results.
Page 85
Results
Reporting
The system is designed to
identify potential
disagreements between
expert reviewers and to
enable resolution of each
disagreement by a
designated review team
leader.
Projected results are
provided after each
sampling round.
The Reports page
provides the audit trail of
each and every predictive
coding project for any
single Relativity project.
The end product is a gold
standard control set of
samples designed to
minimize the effects of
unintended consequences.
The choices that follow the
results are to (1) provide
the system with more
training, (2) validate the
results, or (3) bulk code
the results within the
Relativity database.
The results arm the case
team with the appropriate
information to inform next
steps in the process.
The bulk-coded results are
always associated with
their specific project. This
empowers the case team
to use them for further
review, production, witness
prep, etc.
IIA / ISACA Orange County Chapters
Ernst & Young's predictive coding
multi topic modeling
► Hundreds
►
►A
of topics can be derived from any dataset
e.g., coffee, trade, money, earn, ship, …
topic or set of topics is associated with each document in a dataset
►
A single document is associated with one or more topic
►A
unique probability score is assigned to each topic on a
document-by-document basis
►
Document 1: coffee .23, money .17, trade .02, …
►
Document 2: ship .35, coffee .22, earn .12, …
► The
system predicts a result when the distribution of topics
associated with a single document corresponds to the properties
of one of the coding decisions
Page 86
IIA / ISACA Orange County Chapters
Ernst & Young's predictive coding
multi-topic example
Topics
107
Concepts
raptor:
swap:
balance:
asset:
harrier:
condor:
talon:
Scores
0.0505366246411
0.0102291363833
0.00898489388056
0.00448260133491
0.0031886556012
0.00296781198298
0.002835555109413
185
swap:
raptor:
hedged:
jedi:
hedge:
hedges:
balance:
0.0514520199598
0.0308784376651
0.0078593372283
0.00610262728228
0.00581558674364
0.00245189742511
0.00179358246048
236
swaps:
hedge:
hedges:
0.0333760267552
0.0319537790458
0.0109633042564
390
security:
asset:
underlying:
0.00944631119958
0.00668004462427
0.00281486364346
calpers:
0.000972035033758
377
Page 87
Document
From: Scott Sefton
Sent: Wednesday, August 30, 2000 8:38 PM (GMT)
To: Julia Murray
Cc: Gordon McKillop; Ryan Siurek; Trushar Patel;
Stuart Zisman; Mary Cook; Sara
Subject: Raptor Hedges
Topic
proportions
and
assignments
I thought I'd communicate to the group some recent
developments.
1. Gordon and Ryan advise that the swaps for
assets in JEDI 1 and 2 will cover our
proportionate economic interest in the asset
within JEDI. Gordon believes the spreadsheet
of proposed swaps reflects this. The swap will
be between Harrier and Talon and there will be
no back-to-back swap with JEDI. There should
be no required disclosures to CalPERS
resulting from these hedges.
2. Gordon and Ryan advise that for assets
currently on our balance sheet that are hedged
with Raptor and then sold to Condor, there
will be no reduction in the hedge or any backto-back hedge when it moves into Condor. The
swap will be between Harrier
and Talon.
3. Mark Taylor and Bob Baird have concluded that
the swaps can be terminable by Harrier if the
underlying security is sold.
IIA / ISACA Orange County Chapters
► Each
topic is a
distribution over
words
► Each document
is a mixture of
corpus-wide topics
► Each word is drawn
from one of those
topics
Methodological overview
►
Concept induction
►
►
Fact pattern analysis
►
►
Emotive tone detection, dialogue act classification
Social network and actor analysis
►
►
Fact extraction, workflow analysis, coherence assessment
Linguistic analysis
►
►
Ontologies, concept mining, entity extraction
Internal and external
Predictive modeling techniques
►
Page 88
Supervised and unsupervised
IIA / ISACA Orange County Chapters
Concept mining
►
Automatic detection of:
►
►
►
►
Jargon
Abbreviations and acronyms
Euphemisms
Examples
►
►
►
►
Page 89
NCM is involved as the clmt is looking at his 4th Sx
He hopes to RTW for the INS if his sx will decrease
RTW Plan: NCM will facilitate appropriate surgical tx
Clmt has not RTW in any capacity at any time since DOI
IIA / ISACA Orange County Chapters
The Future of eDiscovery:
Where to From Here
Take a PROACTIVE APPROACH to your ESI
►
Reduce Risk – Keep your organization OFF the Front Page of
the Wall Street Journal
►
►
►
►
►
Page 90
Understand your Network
Create a Data Map
Define Policies and Procedures
Make friends with the other Departments!
Coordinated global eDiscovery Management
IIA / ISACA Orange County Chapters
The Future of eDiscovery:
Where to From Here
►
Understand Major Corporate Challenges:
Your Network!
►
►
►
►
Dispersed network of unmanaged, unstructured, semi-structured and
structured data.
ESI Search and Collection software is limited by the size of your pipes
and the power of your hardware.
eDiscovery in the Cloud?
Privacy Laws – for organizations with Global reach
►
►
►
Page 91
EU Data Privacy Laws
German Works Councils
France – they will throw you in jail!
IIA / ISACA Orange County Chapters
The Future of eDiscovery:
Where to From Here
Conduct Early Case Assessment
►
IT: Involves project scoping from a data collection/processing stance
►
►
►
►
Legal: An early assessment of the issues they care most about
►
►
►
►
How best to be able to demonstrate that their process is “defensible” (reasonable)
How best to put a litigation hold in place and accomplish defensible preservation
Potential strategic advantages, or cards to play, when at meet-and-confer stage
Important to document advanced thinking that went into choosing
►
►
►
where does the data reside
what sources are going to be challenging to collect
what is the volume of data that needs to be collected
Which custodians should be targeted for preservation
What search criteria should be used for collecting ESI from those custodians for
preservation?
Need advanced testing/sampling
Page 92
IIA / ISACA Orange County Chapters
The Future of eDiscovery:
Where to From Here
Get to Know the Special Masters in Civil Litigation
►
►
►
Special Masters are appointed by the courts under Rule 53, Federal
Rules of Civil Procedure, to act as assistants to the court for the
supervision of eDiscovery related issues.
Fill the competency void created by one or all of the other actors in
the dispute resolution process: lawyers, litigants, and judges.
Four different roles for e-Discovery Special Masters:
►
►
►
►
Page 93
facilitating the electronic discovery process;
monitoring discovery compliance related to ESI;
adjudicating legal disputes related to ESI; and
adjudicating technical disputes and assisting with compliance on
technical matters, such as conducting computer/system inspections.
IIA / ISACA Orange County Chapters
Thank you
Matthew L. Miller, Esq.
Assurance Services
Fraud Investigation & Dispute Services
Matthew.Miller@ey.com
212.518.4204
Page 94
IIA / ISACA Orange County Chapters