The Model is Only as Good as its
Inputs!
A robust framework for ensuring reliable
input data- including qualitative and
quantitative tools and a step by step
approach for validating information.
2008 Palisade Risk and Decision Analysis
Conference
Scott Randall, Lloyd’s Register North
America
November 14, 2008, Jersey City, NJ
RISK MANAGEMENT SERVICES
Lloyd’s Register Group
• Lloyd’s Register: over 248 years of risk & uncertainty management
consulting
• 6000 people, 150 offices in over 80 countries
• providing Certification, Verification and Validation of designs,
management systems, projects, strategies
• When you want assurance, think of Lloyd’s Register, because
we believe that Life (and everything related to its
sustainability) Matters.
RISK MANAGEMENT SERVICES
Safety Minute: Validating the Signs of a Stroke
• What is a stroke?
• What are the signs of a stroke?
• How can you validate the signs of a stroke?
RISK MANAGEMENT SERVICES
Safety Tip: Validating the Signs of a Stroke
You could save a life- if you simply remember the first three letters of the word
STROKE and understand what they mean.
If you suspect a stroke, make three simple requests:
Stroke – Ask the individual to Smile.
STroke – Ask the person to Talk. Say a simple sentence (“It’s a sunny day”).
StRoke - Ask the person to Raise both arms.
If he or she has trouble with ANY ONE of these tasks, call the emergency
services immediately, describe the symptoms.
RISK MANAGEMENT SERVICES
Background and Conclusions
• Public companies today are suffering from a crisis of confidence
from their stakeholders because enterprise level risks are not
being effectively managed.
• Enterprise level risk management requires good information
about the market and about stakeholder expectations.
• Information gathered is often poor-contributing to the
stakeholder crisis of confidence.
• Managing stakeholder expectations and enterprise risks using
reliable information achieves sustained competitive advantage.
RISK MANAGEMENT SERVICES
Agenda
1. Reliable information requires a systematic approach involving:
• Framework
• Process
• Tools
2. The framework and process are briefly presented followed by a
demonstration of 3 examples using some basic tools/techniques
3. Stakeholder confidence is shown as the link between reliable
information and competitive advantage.
For publicly traded companies, who are these stakeholders and what are their
roles?
RISK MANAGEMENT SERVICES
A Public Company Stakeholder Pyramid
RISK MANAGEMENT SERVICES
Stakeholder Analysis Template
RISK MANAGEMENT SERVICES
Typical Oil & Gas Company Stakeholder Analysis
Results
(See table 1 on page 2 of Paper for a Summary)
RISK MANAGEMENT SERVICES
Today, all Stakeholders are from Missouri !
RISK MANAGEMENT SERVICES
Some Stakeholders have begun to take this on themselves
(i.e. Credit Rating Agencies)
• Standard & Poors approach to Due Diligence for Enterprise Risk:
(PIM)*
• Policy
• Infrastructure
• Methodology
* Samanta, P., Enterprise Risk Management And Risk Assessment, Ratings Direct, Standard & Poors Inc., November 18,
2005, New York, NY
RISK MANAGEMENT SERVICES
Data Validation and Competitive Advantage
•Data Validation is part of the Broader Discipline of Information
Integrity
•Information Integrity is a cornerstone of the emerging field of
Enterprise Risk Management
•Enterprise Risk Management can help companies create a
sustainable competitive advantage.
Specifically, how can better data validation lead to a competitive
advantage?
RISK MANAGEMENT SERVICES
Due Diligence for Information Integrity
RISK MANAGEMENT SERVICES
The Framework for Information Integrity
Five Fundamentals
of Information Integrity
Criticality
Consistency
Accuracy
Transparency
.
Competence
RISK MANAGEMENT SERVICES
How Important is it?
How Precise is it?
How Accurate is it?
How Traceable is it?
Do the people gathering it know what they
are doing?
A Three Phase Process For Information Integrity
RISK MANAGEMENT SERVICES
The Enterprise Risk/Information Integrity
Toolbox
(Circled
Tools
Illustrated
Today)
RISK MANAGEMENT SERVICES
Validation of secondary information should occur
between the Discovery and Assessment Phases of the
Process
Secondary Validation Steps
RISK MANAGEMENT SERVICES
Validation Steps and Examples
Basic Validation
Initial Data Screening
Current Data
Original (vs
Repackaged)
Data
More than one
source used
a
Qu
Qu
a
lita
e
tiv
nti
ta
Systematic Data
Collection
Disciplined and
Unbiased Process
Variance Analysis and/
or Outlier Rules
Developed
Visual Comparison of
data sets
Advanced Validation
Use of an Independent
Technical Assesor
Second or Third-party
verification/sign-off of
dataset
Verified
Structured Logic
tiv
e
Petroleum Reserves
Example
RISK MANAGEMENT SERVICES
Quantitative Data
Model Used
Historical
Backtesting
Performed
Production cost
example
Reasonable
Certainty achieved
via simulation
Corrosion rate
example
Example: the use of one basic validation tool
ali
Qu
Qu
a
iv e
tat
nti
tat
ive
Pressure Vs. Production Rate
45
40
35
P ro du ction ra te S TB / D
30
25
20
15
10
35
40
45
50
55
60
65
Pressure
RISK MANAGEMENT SERVICES
70
75
Petroleum Reserves Estimating
A. Proven undeveloped reserves (PUDs) are
reported to investors & analysts and affect
stock oil and gas company stock prices.
B. A company geologist’s bonus is sometimes
based on his discovering oil and gas!
• Sometimes the combination of A + B
makes us question, how proven is
“proven”!
• This leads to a loss of stakeholder
confidence.
San Felipe Resources and the tale of the missing reserves………
RISK MANAGEMENT SERVICES
Data Table-Historical Data and Future Projections*
SCENARIO
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
RKR04
FIRST_PROD WELL COUNT
30-Jun-01
1
31-Jul-01
2
30-Sep-01
3
30-Sep-01
4
30-Nov-01
5
21-Dec-01
6
31-Dec-01
7
31-Dec-01
8
31-Dec-01
9
20-Jan-02
10
24-Jan-02
11
14-Feb-02
12
21-Feb-02
13
21-Mar-02
14
01-Apr-02
15
03-Apr-02
16
26-Apr-02
17
24-May-02
18
03-Jun-02
19
04-Jul-02
20
23-Oct-02
21
27-Oct-02
22
25-Dec-02
23
13-Mar-03
24
03-Apr-03
25
01-Sep-03
26
27
28
29
30
Historic Ultimates
Future Drilling
20,686
19,215
24,527
5,731
13,272
11,749
216
16,385
2,462
4,923
8,418
12,123
9,445
4,863
3,714
7,550
6,648
5,879
5,582
3,297
2,650
2,994
55
1,582
671
649
3000
3000
6200
6000
* Volumes in MMSCF
RISK MANAGEMENT SERVICES
Ultimate Recovery
(MMCF)
A Simple Scatter Plot Raised Some Difficult
Questions
RISK MANAGEMENT SERVICES
Examples: use of more advanced validation tools
45
40
35
30
25
20
15
10
75
70
65
60
55
50
45
40
35
Historical Comparisons, Linear Regressions, @Risk
RISK MANAGEMENT SERVICES
Corrosion Rate Example
Background:
• Calculating pressure vessel leaks from corrosion not trivial:
- Typical refinery/chemical plant has over 3,000 piping loops,
pressure vessels or heat exchangers with hazardous, toxic and
corrosive fluids
• Standard Practice* is to calculate corrosion based on change
in wall thickness measurements over time using straight line
between 2 points:
Long Term Corrosion Rate:
tinitial-tactual/ #years between readings
and
Short Term Corrosion Rate:
tprevious-tactual/# years between readings
*API 510, Pressure Vessel Inspection Code: In-Service Inspection, Rating, Repair, and Alteration, 9th ed.
RISK MANAGEMENT SERVICES
Process Plant-Pressure Vessel Wall Thickness
(Measurements over 5 Years)
Measurement
Date
11/1/1998
11/1/1998
11/1/1998
11/1/1998
11/17/1999
11/17/1999
11/17/1999
11/17/1999
11/19/2001
11/19/2001
11/19/2001
11/19/2001
7/29/2002
7/29/2002
7/29/2002
7/29/2002
6/30/2003
6/30/2003
Thickness
Measurement
1.017
1.026
1.016
1.026
0.994
1.008
0.992
1.009
0.864
1.004
0.964
1.005
0.989
1.000
0.990
1.003
0.994
0.984
RISK MANAGEMENT SERVICES
Measurement
Location
Head (W)-1
Head (W)-2
Head (E)-1
Head (E)-2
Head (W)-1
Head (W)-2
Head (E)-1
Head (E)-2
Head (W)-1
Head (W)-2
Head (E)-1
Head (E)-2
Head (W)-1
Head (W)-2
Head (E)-1
Head (E)-2
Head (W)
Head (E)
Year
0
0
0
0
1
1
1
1
3
3
3
3
4
4
4
4
5
5
Long & Short Term Corrosion rates
per Code = app. 6.5 mills/yr…….
…….but we questioned
the accuracy of the
input data!
Establishing Confidence in Calculation Using
Simulation
+ @RISK + TREND SUMMARY
Use these to:
Results of Linear Regression of
Thickness Measurements:
ƒ corrosion rate (slope of least squares
line)=7.1 mills/yr
but r2 (.13) was terrible!
RISK MANAGEMENT SERVICES
• Isolate bad data
• Give a certain level of
confidence around our final
corrosion rate
Summary Trend of Wall Thickness Measurements
isolates bad data
Head (W)-1_1998 to Head (E)_2003
Thickness Measurements
1.2
1.15
1.1
5% - 95%
1.05
1
+/- 1 Std. Dev.
0.95
Mean
0.9
0.85
0.8
0.75
RISK MANAGEMENT SERVICES
Head (W)_2003
Head (E)-1_2002
Head (W)-1_2002
Head (E)-1_2001
Head (W)-1_2001
Head (E)-1_1999
Head (W)-1_1999
Head(E)-1,1998
Head (W)-1_1998
0.7
Scatter Plot highlights the same problem.
Meaured Thickness for Vessel 102PV17102
(Nominal Thickness 0.938 in)
1.04
Measured Thickness (inches)
1.02
1.00
0.98
0.96
0.94
0.92
0.90
0.88
0.86
0.84
7/1998
12/1999
4/2001
9/2002
1/2004
Elimination of this piece of data improved confidence around the
corrosion rate projection
RISK MANAGEMENT SERVICES
Michael Porter and Competitive Advantage*
* Porter, M.E., Millar, V.E., "How Information Gives You Competitive
Advantage", Harvard Business Review, July-August 1985
RISK MANAGEMENT SERVICES
For Modern Asset Intensive Industries, Porter
Gets it Right (Almost)
• An Industry can still be described by the relative power of 5
Forces
• Competitive Advantage still comes in two forms
• Information Technology (IT) is key
• But not IT as we know it!
• A company’s “Value Chain” is an important way to see how
information and Competitive Advantage are related
21st Century Modifications
• The Enterprise Value Chain Map illustrates Enterprise Risks*
• The Integrity (Quality) of Information is more important than its
aggregation and manipulation
• The Powerful Stakeholders are the kings
* Not dealt with here
RISK MANAGEMENT SERVICES
Modified Porter Model for 21st Century
Competitive Advantage*
Porter’s Five Forces of
Industry Attractiveness
Enterprise Risk
Management/
Reliable Information
1. New
Entrants
2. Powerful
Suppliers
3.Substitute
Products
4.Buyer
Power
Competitive
Advantage
Strategic Vehicle
External Information Integrity Toolbox
* Identify Strategic
Issues along the
Value Chain
* Focus on the
Stakeholders
5. Competitor
Rivalry
* Spot
Opportunities
Improved
Management
Decisions
* Reduce
Uncertainty
* Manage Major
Risks
Increased
Stakeholder
Assurance
* Lowered Costs
* Enhanced
Differentiation
Randall,S., Energy, Risk and Competitive
Advantage: The Information Imperative, PennWell, 2008.
RISK MANAGEMENT SERVICES
Addressing stakeholder needs through a Framework,
Process and a Toolbox for ERM and Information Integrity
Information Integrity
Approach
Stakeholders and
Responsible
Parties
Level of Control
Five Fundamentals
of Information Integrity
Oversight
Criticality
Reasonable
Assurance
Consistency
Accuracy
Transparency
.
Competence
The Information Integrity
Toolbox
Governance
Strategy &
Management
Implementation
RISK MANAGEMENT SERVICES
Fundamental
Principles
Internal Controls
Policies
Procedures
Tools
Techniques
Methodologies
Major Individual Investors
Institutional Investors
Creditors
Regulators
Boards of Directors
Credit & Stock
Analysts
External Auditors
Executive & Upper
Management
Internal Auditors
Middle Managers
Supervisors
Internal Analysts
How Data Validation Can Lead to Competitive Advantage
The Obvious
•
Better information leads to better management decisions
The Not so Obvious
•
A more robust approach to information integrity provides assurance to your
stakeholders that you can meet or exceed their expectations
•
Fewer regulatory burdens (costs)
•
Lower cost of capital due to lower “idiosyncratic risk” assigned to your
company by rating agencies and investors
RISK MANAGEMENT SERVICES
The Lloyd’s Register Group
works to enhance safety and
approve assets and systems at
sea, on land and in the air –
because life matters.
scott.randall@lr.org
+832.298.4380
RISK MANAGEMENT SERVICES
Questions & Comments
RISK MANAGEMENT SERVICES