Granular Metrics,
Feedback and Experimentation
Arnoldo Hax
Alfred P. Sloan Professor of Management
Contributions of the Delta Model
The Triangle
Adaptive
Processes
Opening the
mindset to new
strategic
position
The Best
Three Strategic Options:
Product does
z Best Product
not always win
Linking
Strategy with
Execution
Execution is
not the
problem,
linking to
strategy is
Total Customer Solutions
z System Lock-in
z
Execution is captured through Three
Adaptive Processes:
Operational Effectiveness
z Customer Targeting
z Innovation
z
whose roles need to change to achieve
different Strategic Positions
Aggregate
Metrics
Granular
Metrics and
Feedback
Measuring
success
Discovering
performance
drivers
Good
financials do
not always
lead to good
results
Aggregate performance metrics need to
reflect each of the Adaptive Processes
and their role based upon the strategic
position
Managing by
averages
leads to below
average
performance
Business is non-linear. Performance,
particularly bonding, is concentrated.
Granular metrics allow us to focus on
underlying performance drivers, to
detect variability, to explain, to learn, and
to act.
Product performance
z Customer performance
z Complementor performance
Business
is non-linear. Performance,
z
The Delta Model and Granular Metrics
Metrics
Select Performance Indicators
Act from Variability
• Define program
• Define structured tests
• Rollout
Learn from Variability
• Hypothesize cause and
effect
• Evaluation
Detect Variability
• Drill down to detailed
segmentation
• Isolate variability
Explain Variability
• Identify performance
drivers
• Correlate drivers with
variability
Drivers of variability for selected performance indicators
Performance indicators
Drivers of variability
Product cost and quality
z
Customer profit
z
Complementor contribution
z
Business segment economic
value
z
Scale
z DensityDensity-e.g. concentrat
concentration of serv
service
ice
z Location
cation
z Labor productivityproductivity- practices and training
training
z Equipmen
Equipment product
productivity
ivity-- design
design
z Process design
z And
And so on
Customer size
z Customer revenue
z Tenure
z Acquisition cost
z Channel
Channel mix
z Customer care supp
support
z Customer investments in relation
relationshi
ships
z And
And so on
Size of releva
relevant complementor products
z Complementor investment in bu
business
z Relative size in customer value chain
z Complementor contributio
o customer economics
contribution tto
z Exclusivity of relationship
relationship
z And
And so on
ROI (return on
on investment) – profitability
z RiskRisk- volatility and covariance
covariance
z Option value
z Investment base
z Cashflows
Cashflows
Experimentation as the basis for effective change
change
Large
Size of
Change
Small
The middle road:
lower returns, or
unacceptable when
first mover
advantage is high
Unacceptable risk,
risk,
as a starting point.
point.
Highly desirable as
as
endpoint
endpoint
Ineffective: lower
returns, or
unacceptable when
first mover
advantage is high
Testing: the
relevant area for
experiments
leading to large
change
Slow
Speed of
Change
Fast
Cost De-Averaging:
Using Granular Metrics To
Drive Performance
1. The Value Chain of the Local Business Data Circuit
Switching
Transmission
Order
fulfillment
Etc.
Marketing
Billing
Average Cost = $395/order
3. The 80-20 Cost Effect
100
80
% of costs
$1,000
Cost/order
2. Individual Order Cost
60
40
20
$100
Most
Expensive
Orders
Least
Expensive
0
20
40
60
% of orders
THE BEHAVIOR OF COST - THE CASE OF BUSINESS DATA SERVICES
Figure by MIT OCW.
80
100
The drivers of the cost of order-fulfillment
fulfillment
Cost of order
fulfillment
Location of
work centers
Activities of
order fulfillment
Manpower
productivity
Equipment
Order Fulfillment
Headcount required to process
1,000 orders per month
70
Competitor 1 - Houston
Company
Competitors
Competitor 1 - Chicago
Chicago
}
39%
gap
1
1,000
Client performance
curve
New York
Tulsa
Charlotte
Competitor 1 ­
Denver
Kansas City
Competitor 1 ­
San Francisco
Competitor 2 ­
San Jose
Competitor 2 Indianapolis
2,000
Best-in-class
performance
5,000
12,000
27,000
Scale = 72%
Log/log graph
Competitor 3 ­
Jacksonville
60,000
Orders/Month
T HE BE H AV I O R O F CO S T BY L O CAT I O N
Figure by MIT OCW.
Customer
ready
'Defect-free Path'
Automatic
switch map
Fast
facilities test
$100 (per
access leg)
Valid design
Facilities
available
Order
Facilities
not
available
Access
available
Pass test
Access
not
available
10x cost
difference
Access
available
Access
not
available
Skin test
Invalid design
'Defect-prone Path'
Manual
switch map
Fail
facilities
test
Fail access
test
THE BEHAVIOR O F C O ST B Y FIV E A C TI V I TIES IN TH E VAL UE CHAI N
Figure by MIT OCW.
$2,000 (per
access leg)
Customer
not ready
1 MILLION POSSIBLE ORDER PATHS
Pass access
test
The behavior of cost by type of defects
defects
Client
Defect-free
Business Data
Service Orders
% of
orders
Cost/
order
Averag
e cycle
time
71%
$100
5 Days
18%
24.0%
$900
25 Days
60%
Total
cost
Focus of
program
developmen
t
Business Data
Service Orders
Business Data
Service Orders
5.0%
$1,800
45 Days
Average
1
$395
12.3 Days
22%
Craftsmen Efficiency
Total hours/cleared fault
10
8
6
S:1
4
2
0
A
B
C D
E
F G H I
Craftsmen
J
K
L M
Fault not fixed correctly
& repeats
Pair throw: reroute around
fault
Request assistance
Refer fault to someone else
(specialist)
Fix fault
N
Cumulative cost (%)
100
80
The Need for
Granular Metrics
0
0
25
100
Cumulative craftsmen (%)
THE BEHAVIOR OF COST BY INDIVIDUAL WORKERS
Figure by MIT OCW.
Records Errors Concentration
100
100
80
80
% of Records Errors
% of Line Faults
Line Fault Concentration
100% of demand pair
reclamations are
concentrated in 17%
of junctions
60
40
20
0
0
20
40
60
80
100% of records
errors are concentrated
in 19% of equipment
60
40
20
100
% of Junctions
0
0
20
40
60
80
% of Equipment
THE BEHAVIOR OF COST BY TYPE OF EQUIPMENT
Figure by MIT OCW.
100
Cost/Unit
Benchmarks
Cost Drivers
Best Practice Sessions
Select
Associates
Director/Process Reviews
Standard
Supervisor
a b
Individual Performance
Individual Reviews
Standard
a b c
Supervisor
Supervisor Review
Ind. Actual Target
Total
District/Center Reviews
Supervisor
Total
Actual Target
'Supervisor Coaching'
OPERATIONAL EFFECTIVENESS: ADAPTIVE FEEDBACK USING GRANULAR METRICS
Figure by MIT OCW.
Profit De-Averaging:
Using Granular Metrics To
Drive Performance
THE NEED FOR GRANULAR METRICS
3x
2x
35
Profit Margin
1x
0
-x
Cumulative percentage of customers
10
20
30
40
50
60
70
80
90
100
-2x
85% of Profits
46% of Profits
40% of Profits
-3x
21% of Profits
-4x
-5x
-6x
-7x
P RO FIT MA RG IN CO N T RIBU T ION
Figure by MIT OCW.
The drivers of customer profitability in the credit card business
s
Profit by customer
Revenue per
customer
Number of
customers
Balance level
per month
Months of
tenure
Monthly Profit / Customer
75th Percentile +
1.5* Box Size ('Upper
limit of connected data')
$70
$60
$50
$40
$30
$20
$10
$0
$(10)
$(20)
75th Percentile
Median
25th Percentile
$10-$100
$100-$200
$200-$400
$400+
Revenue/Customer
25th Percentile -1.5*
Box Size ('Lower limit
of connected data')
VA R I A B I L I T Y O F P R O F I T S B Y U S A G E L E V E L S
Figure by MIT OCW.
Profit Margin (ROS)
Company Scale
18%
Customer Share
22%
15%
5MM
10MM
20MM
30MM
Number of Customers
30
20
10
0
-10
-20
-30
-40
-50
$1000 $2000 $3000 $4000
Balance Level $/Month
Customer Tenure
30
20
10
0
-10
-20
-30
-40
-50
10
Months of Tenure
FURTHER E X A MI N AT I ON O F P R O F I T DR IVER S
Figure by MIT OCW.
20
30
Cumulative Profit
5x
}
}
Tailor offers to individual customers
(i.e., increase response & profit with differential
offers tailored to customer's preferences)
3.5x
x
0
Offer one product to customers, based on
both response & profitability
(i.e., acquire customers likely to have good volume/
margin/lifetime characteristics)
}
Offer one product to those who will respond
(i.e., maximize response for a given offer, assuming average economics)
150,000
300,000
Cumulative Number of Customers Solicited
T H E E F F E C T S O F C U S TO M E R TA R G E T I N G
Figure by MIT OCW.
}
Offer one product to
everyone
Untargeted solicitations
yield a negative profit
Customers’ targeting behaviors
behaviors
Behaviors
Typical competitors
Capital One
Offer
z Test/year
z Service tier
z Key metric
z Targets
z Solicitations/year
z Segmentation frequency
z Risk orientation
z Data sources
z Staff analysts
75 products
30
4
ROE, Delinquency
High response likelihood
50 million
Biannual
Risk averse
80
20
5,000 products
15,000
13
NPV
High NPV potential
400 million
Ongoing
Price for risk
50,000
150
z
Capital One- achieving a Total Customer Solutions through
Customer Targeting
Variation 2
Variation 1
Select
Offers
Further
improvement
Offer
Creation
Core offer
• Interest
Interest rates
rates
• Fees
• Promotions
Mass
marketing
Screen for
profitability
• Full life
cycle customer
profitability
Variation 2
Variation 1
Test cells
Collect
customer
response
Success metrics
metrics
Customer NPV
z Acquisition cost
z Activation
z Percent with revolving balances
z Fee income per account
z Charge-off per account
z Tenure
z
Typical competitor
Capital One
$62
$77
58%
25%
$55
$71
4 years
$427
$67
79%
40%
$73
$54
6 years
Value Created by Business Unit
% of Total Company Value Created
60
100% of
value
created
20
42% more value
created
-21% of value created
-21% of value created
i) 20% of new
capital
ii) 32% of R&D
i) 36% of new capital
ii) 25% of R&D
i) 20% of new capital
ii) 22% of R&D
15
10
5
0
i) 24% of
new
capital
ii) 21% of
R&D
-5
-20
-40
0
25
50
% of Equity Investment
75
100
1) 6 BUs contribute 100% of value created
2) 16 more BUs contribute another 42% of value created
3) Those 22 BUs are worth $ 15.5 billion, on an equity investment of only $4.2 billion
4) 44% of new capital investment and 53% of R&D spending over the next three years is going to BUs
earning economic profits
5) 56% of new capital investment and 47% of R&D spending are going to BUs that will be generating
economic losses
SOURCES OF VALUE CREATION
Figure by MIT OCW.