1
Test & Learn
as a Tool to Optimize
Call Center Performance
Stephanie Denny
SVP & Inbound Channel Sales Manager
Bank of America Card Services
Test and Learn:
The Practical Use of a Six Sigma Tool
• Opportunity: Optimize the sales process in our
Credit Card Customer Service call centers
• Challenge: Use the same “test and learn”
discipline in the inbound call channel as we do in
direct mail marketing
3
Why Model After Direct Mail?
• Direct mail measures everything
– Offer, message, creative and response rates
– Uses a Champion-Challenger approach to
continually improve response rates
• It uses a disciplined process of
measurement from beginning to end
4
Our Challenge
• Existing call center desktops are designed
for customer service
• Sales engine -- limited measurement
capabilities
• We needed a different way to test, separate
from a production environment
5
Enter Design of Experiment
• What is DOE?
– An engineering tool, originally used in
manufacturing environments
– It’s a way to efficiently test many variables
within a process, and still get statistically valid
results
– DOE tests many variables, with the objective to
reduce variability
• So how could we use it to build an effective
sales process?
6
We Used DOE to Design Sales Tests
• DOE is a way to test many variables, or attributes,
at the same time
• Within each attribute, there may be different
variations, or levels, to be tested
Variable
Offer
Message
Rebuttal
Level 1
Promotional Rate
Account Feature
Rising Rates
Level 2
Contractual Rate
Simplifies
Valued Customer
• For this experiment we have 23 = 8 combinations
Levels
7
Variables
Start with a Hypothesis
• Examples:
– The words we use to offer a balance transfer affect the
customer’s response
– If Contractual Rate balance transfer offers are more
profitable, we should offer them first
– If we give a rebuttal, we will get a better response if we
use the words “valued customer”
• These are all variable factors within a single
customer interaction. How can we efficiently test
all of them with confidence in the results?
8
Setting Up a Test Design
• Below are all the possible combinations of the variables in
the experiment:
Offer
Promotional Rate
Contractual Rate
Promotional Rate
Contractual Rate
Promotional Rate
Contractual Rate
Promotional Rate
Contractual Rate
Message
Account Feature
Account Feature
Account Feature
Account Feature
Simplifies
Simplifies
Simplifies
Simplifies
Rebuttal
Rising Rates
Rising Rates
Valued Customer
Valued Customer
Rising Rates
Rising Rates
Valued Customer
Valued Customer
• Each variable is tested two or more times to see whether or
not it makes a difference in the results
9
Six Sigma Talk
These
are all
the Xs
Offer
Promotional Rate
Contractual Rate
Promotional Rate
Contractual Rate
Promotional Rate
Contractual Rate
Promotional Rate
Contractual Rate
Message
Account Feature
Account Feature
Account Feature
Account Feature
Simplifies
Simplifies
Simplifies
Simplifies
Rebuttal
Rising Rates
Rising Rates
Valued Customer
Valued Customer
Rising Rates
Rising Rates
Valued Customer
Valued Customer
=Y
(Response Rate)
• Design of Experiment looks at the cause-and-effect
relationship between different variables, or Xs
(inputs)
• The output is Y
10
Measurement of the Xs
• DOE enables predictability and optimization
• Our Information Analysis and Modeling team
can build an equation to predict a response,
based on using a combination of the winning
variables
11
And the Winner Is . . .
Results: Balance Transfer Dialogue Test
For demonstration
purposes only – these
were not our actual
response rates.
2.50%
2.00%
1.50%
1.00%
0.50%
0.00%
Account
Features
Simplifies
Control
•With a statistically valid sample size, we can be confident we
have a winning message
12
So How Do We Conduct
Our Tests in the Call Centers?
• We engaged a vendor with Web-based software
designed for this purpose
– Capable of managing complex experimental designs
– Manages many concurrent tests
– Measures each step of the process independently
– Dynamic scripting capability
– Next-day reporting enables us to quickly identify trends
13
We Call Our Experiment “Test & Learn”
• Rapid-cycle testing: many small “test pilots”
running concurrently
• Randomization of components to ensure valid tests
– We measure all parts of the sales process
– We work with our information analysis group to
determine valid sample sizes for each offer
– We continually challenge previous results with new tests
for further improvement
• Objective: Predict and control the optimal settings
for the sales process
14
What is Test & Learn?
• Our Test Lab is a team of Customer Service
associates, the “Test Pilots”
– They have opted to be part of this pilot
– Some of their scorecard metrics are different
• No emphasis on call handle time
• Qualified Good Contacts (QGC) rates are measured
(offers made)
• Their primary metric is for adherence to the tests
15
What Does It Look Like?
All inbound credit card calls
enter VRU
Accounts flagged as marketable
route to Test Lab first
Small percent of
transactions in Test
Lab
All other accounts have
normal routing
Normal service routing
Test Lab call overflow routes
back to general servicing
16
Test & Learn as a
Performance Measurement Tool
•Our Test Lab has enabled
us to measure associate
sales performance
•We could then monitor
and coach efficiently
Conversion Rate
•We combined two metrics
to identify the best and
worst performers
Conversion Rates
are measured as the
ratio of sales to
offers made (QGCs)
VP
Above Average
RC
RCH
GM
VR
AR
KB
QP
YV
JH
Avg. = 2.5%
DR
SB
NP
•Imagine the power of
applying this discipline to
other Customer Service
processes
NM
DM
CB
VV
Below Average
QGC Rate
17
ET
SW
CC
Test & Learn Uses
Six Sigma Discipline
Define
Measure
Analyze
Improve
Control
Design
Experiment
Measure
Each Test
Analyze
Results
Deploy
Learnings
Monitor
Results
•Test & Learn is a continuum in the DMAIC process
•First you learn what to do
–Identify best offers
–Develop optimal environment
•Then you deploy and control
•Then the cycle begins again to enhance the process
18
Key Learnings
• Know Your Metrics: they drive everything
– Metrics are based on the objective of your tests
– Your metrics may be something other than profitability
(such as customer retention rates)
• Your guesses are often wrong – use statistically
valid data to make decisions
• It’s much less costly to fail in a test environment
than in a production environment
19
Consider the Possibilities
• Test & Learn can be a powerful tool to improve
many Customer Service processes
• Control of customer dialogue reduces variability
• Ideas for tests will happen as more associates are
exposed to the process – ask for their ideas
• Test & Learn will become an important part of the
business-as-usual process
20
Contact Information
Stephanie Denny
SVP: Consumer Card Inbound Channel Sales
Phone: 602.597.5656
Fax: 602.597.4725
Email: stephanie.a.denny@bankofamerica.com
Bank of America Card Services
AZ9-502-01-06
1825 E. Buckeye Road
Phoenix, AZ 85034
21