2000 Systems Engineering Capstone Conference • University of Virginia
SCORECARD DEVELOPMENT FOR
A DEBT COLLECTION SYSTEM
Student team: Michael Autuoro, Hope Breskman, John Congable, Arin Shapiro, Rachel Yoo
Faculty Advisor: William T. Scherer
Associate Professor, Department of Systems Engineering
Client Advisor: Douglas Fuller
First Select Corporation
Strategic Operations
4460 Rosewood Drive
P.O. Box 9053
Pleasanton, CA 94566
Doug_Fuller@providian.com
KEYWORDS: agent, gaming, metrics, Promise To
Pay, Refusal To Pay, Right Party Connect
ABSTRACT
First Select Corporation (FSC) is a young
collections company that needed to develop a set of
performance measurements, or metrics, to assist in
operating an effective call center where its primary
business operations take place. The metric
development process started with the first phase of topdown analysis. The team researched to become
acquainted with metrics through previous studies and
case examples. Meetings with FSC management took
place in the early stages to establish overall FSC goals.
Performance measurements that would support those
goals were brainstormed, using knowledge gained from
research.
After the Capstone team received relevant call
center data, the second phase of bottom-up analysis
begun, primarily consisting of data mining and
scorecard development. A scorecard format facilitates
the evaluation process with its graphic layout and
concise, specific, and significant data. The scorecard
was divided into four employee levels within FSC:
management, division, team, and agent.
The scorecard can be traced from the top-level down
to the agent level. Outliers in the management level
scorecard might be detected and further investigated to
determine what behavior might have caused that
phenomenon. Therefore, data from lower levels might
help upper-level management to pinpoint the cause of
the underperformance of FSC as a company.
INTRODUCTION
Providian Financial Corporation is the sixth largest
credit card provider in the world. Providian targets
high-risk customers and, in exchange, charges higher
rates than its competitors. Specifically, Providian
attributes much of its success to its unique collections
strategy. Its excellence in collections led to the
development of First Select Corporation (FSC), a
wholly-owned subsidiary that is completely dedicated
to debt collection. Its primary business operations take
place in its call center located in Louisville, KY. First
Select not only collects Providian customers’ debts but
also buys debts from other institutions at a reduced cost.
In the past few years, Providian has steadily
increased 30-40% in market value. Keeping up with
the rapid growth of Providian, First Select’s call center
in Louisville expanded to over 300 employees in one
year of development. One consequence of this rapid
growth has been the failure to use a standard set of
complete metrics to regularly measure the call center’s
performance.
In order to progress, FSC management, who make
executive decisions, must be capable of evaluating its
performance and knowing which areas require
improvement based on the available data. The purpose
of this Capstone project was to provide an accurate and
useful instrument with which FSC can measure its
performance, target its weaknesses, and exploit its
strengths.
63
Scorecard Development For A Debt Collection System
OVERALL PROCESS REVIEW
The Capstone project was divided into two phases.
The first phase provided an opportunity to get
acquainted with the client, First Select Corporation, and
understand the problem definition. Conceptualization
was the approach to metric development in this topdown phase of the project. In the spring semester, the
Capstone team received data from FSC and was able to
perform data analysis to begin the second phase.
Research took place throughout the entire course of the
project. Particularly valuable was the research about
the process for developing these metrics, the
characteristics of successful metrics, and, in phase two,
the existing measurements of performance for call
centers. Metrics were then generated from the available
data in the second phase, after cleansing and scrubbing
the call center data. The combined efforts of both
phases produced the final deliverable to the client: the
scorecard.
BACKGROUND RESEARCH
The Capstone team chose to use Sprague’s five traits
used as the foundation for metric development during
the project duration. It was important to understand the
value of each characteristic and repercussions of
lacking those traits. When metrics were brainstormed
in the initial phase of the project, the aforementioned
significant attributes were held as guidelines for metric
selection.
SCORECARD DEVELOPMENT
It was established by the FSC Capstone team that a
scorecard would be the most effective means to display
various metrics for each employee level. Scorecards
facilitate efficient and effective performance review.
This was particularly important for higher-level
management so that spotting trouble areas would not be
a time-consuming, arduous task.
The employee structure within FSC can be broken
into the levels shown in Figure 1. A scorecard was
customized for each of the general four levels:
management, division, team, and agent.
Measurements must support the high-level goals.
The overall goals of FSC were used to determine which
metrics would support them. Obviously, goals are
unique to the enterprise and the individuals who make it
up. Therefore, it is universally agreed that metrics must
be unique to the organization they are evaluating. “It’s
a question of using the right metric for the right job”
(Carter, 14).
Research has shown that more and more businesses
are driving improvement through the direction that
measurements of performance can provide. Bob Frost
of Measurement International says, “executives require
significantly better measures to direct strategy and
business performance than ever before” (Frost, 34). All
levels throughout a company can benefit from effective
measures. The challenge is to make the measurement
effective by maintaining high standards during the
metric development process. There are certain vital
attributes a performance measure must have to be
useful. According to Gail Sprague of Vanguard
Communications, there are five main principles for
effective measures:
1. Simplicity
2. Drive the right behavior
3. Measurable and Available
4. Accountability associated with control
5. Support high level business goals
Source: “Three Steps for Creating Effective Call
Center Measures and Reports” by Gail Sprague
64
President
Senior Vice
President
Call center
operations
manager
Dialer
operations
manager
Quality
Assurance
manager
Division
managers
Team
Leaders
Agents
Figure 1. FSC Management Hierarchy
Management Level
In providing top-level executives with information
on call center performance, the following were the
primary objectives:
2000 Systems Engineering Capstone Conference • University of Virginia
1.
2.
The metrics must be succinct and comprehensible.
The metrics must quickly pinpoint abnormalities in
the system.
The metrics must incorporate the experience and
intuition of an executive employee.
All metrics were divided into inbound and outbound
performance due to the distinct differences in the
respective customer characteristics for each.
Although a top priority in metric development was
to find a correlation between potential metrics and
money brought in the door, lack of payment
information precluded this study. With only one month
of summary information, analyses focused on creating
time-series representations of data and recognizing
deviations from normal behavior.
“Division” refers to an account’s status in the
collection process. The data must be divided according
to division because accounts in different divisions are
not comparable to each other. A section manager
oversees all of the accounts that are in his specific
division. Section managers will be most interested in
viewing data on a division level. The same metrics on
the management level scorecard are included for the
division level. If outliers are detected on the
management level scorecard, the division level might
explain some of the distinctive and seemingly peculiar
behavior of a single division.
3.
As a first step in identifying metrics, several
interviews with top-level executives took place. At
these meetings, call center information was prioritized
and each executive’s preferred method of analysis was
reviewed. This information led to a set of preliminary
metrics and relationships that were aligned with the
intuition and expertise of the leaders of the company.
In conducting this set of interviews, information was
obtained that could not be garnered otherwise. For
example, several executives mentioned that they often
gauge the quality of a day by calculating the ratio of
Promise to Pays to Total Dialer Hours. Historically,
this ratio has had little variance from day to day and
therefore, can be used to evaluate a day’s performance.
The second stage in the development of the toplevel metrics was to verify and validate their usefulness.
In doing so, the Capstone team generated specific call
center scenarios and rated each metric on how well it
indicated that scenario. Specifically, call-center
technological problems, dialer malfunction, and
employee gaming situations were hypothesized. In
rating each metric on how well it predicted the
aforementioned scenarios, the scorecard was further
developed.
As a result of an additional iteration of this twostage process the Capstone team completed the toplevel scorecard, which consists of the following
metrics:
1.
2.
3.
4.
5.
Total number of Promise to Pays achieved per day
Total number of Dialer Hours per day
The number of Right Party Connects/Dialer Hour
per day.
The percentage of Right Party Connects that were
Promises to Pay, per day.
The percentage of Inbound Calls that are
abandoned by customers in the queue.
Division Level
Most of the data in the division level is presented in
a time series format so that the section managers can
view trends and patterns in the data, as well as
individual anomalies. In addition, each division has a
summary graph that shows how his division is
performing compared to the others. All five of the
high-level metrics are of interest at the division level.
Team Level
The team level of the scorecard compares different
teams in the same business aspect of First Select
Corporation, i.e. First Select Front End. For our
template scorecard, teams were randomly assigned and
varied in size from ten to fifteen agents. Similar to the
relationship between the management and division
level, outliers detected on the division level might be
explained by performance on the team level within a
particular division. The team level is utilized by
supervising managers when comparing the productivity
of different teams of agents. For example, if a team is
under-performing, the unit manager may notify that
team’s leader. Metrics used for the team level are the
same as those used on the individual agent level, just
average for the team.
The identification and acknowledgement of team
success is critical in business today. The team level
scorecard denotes these qualities and generates a
greater sense of teamwork in the office. In addition to
their own success, agents might be encouraged to focus
on the success of the team. This approach might reduce
the existence of a competitive atmosphere, thus
reducing the chances of cheating. Having members
65
Scorecard Development For A Debt Collection System
working and measured as a team will promote morale
and reward those groups of agents that have the same
goals as the management of First Select Corporation.
Agent Level
While it is important for FSC to focus on teamwork
and avoid individual competition, evaluation on the
agent level is a necessity for numerous reasons. Being
the bottom rung of our scorecard, the agent level can be
used by team and unit managers to compare an agent’s
performance to that of their peers. Knowing individual
agent performance will allow First Select to isolate the
superior employees from the weaker ones. Consistently
low performers might be given more training for future
improvement, while strong agents might be given more
responsibility or aid the struggling representatives.
developed by the FSC Capstone team, using % PTPs in
Outbound Calls as the measurement of performance. %
PTPs are the percentage of connected calls in which the
customer promised the agent a certain payment amount
to contribute to the debt owed.
When an FSC employee on the management level
receives his scorecard, he will first look for unusual
behavior, according to what has already been
established as the norm (typically the average value).
For example, a graph of % PTPs for all divisions for
a three-week period will be displayed (each week is
connected by a line):
)dnuobtuO(PTP %
%0.43
The metrics used in evaluating the agent level were
the following:
%0.23
%0.03
Promises to Pay per Right Party Connect
Refusals to Pay per Right Party Connect (RPC)
Talk Times per time logged onto the dialer
Update Times per time logged onto the dialer
Outbound Right Party Connects per Dialer Hour
(RPC/DH)
%0.82
%0.62
%0.42
%0.22
%0.02
0 0/ 8 / 1
0 0/ 9 / 1
0 0/ 0 1 / 1
0 0/ 1 1 / 1
0 0/ 2 1 / 1
0 0/ 3 1 / 1
0 0/ 4 1 / 1
0 0/ 5 1 / 1
0 0/ 6 1 / 1
0 0/ 7 1 / 1
0 0/ 8 1 / 1
0 0/ 9 1 / 1
0 0/ 0 2 / 1
0 0/ 1 2 / 1
0 0/ 2 2 / 1
0 0/ 3 2 / 1
0 0/ 4 2 / 1
66
0 0/ 5 2 / 1
The following is a step-by-step application of the
drilldown functionality of the scorecard for FSC
0 0/ 6 2 / 1
DRILLDOWN APPLICATION
0 0/ 7 2 / 1
Although each metric was analyzed and presented
individually to compare agents to one another, a star
chart was also developed to provide an extensive
summary for each agent shown. Agents might quickly
realize what metrics by which they are being measured,
and thus manipulate, or game, the system to give the
appearance of better performance. Therefore, the
ability to look at numerous metrics in one step in a star
chart provides First Select with the ability to detect this
gaming phenomenon, target weaknesses, and exploit
strengths.
0 0/ 8 2 / 1
Since there is limited data regarding the dollars
collected by each agent, PTPs are the best surrogate to
measure each agent’s contribution to the dollars
brought in the door. In the case of RPC/DH, only
outbound performance was valid under the assumption
that almost all inbound calls are RPCs and an agent has
virtually no control over the number of outbound RPCs
he or she works.
etaD
Graph 1. % PTP from Outbound Calls
The reader sees that the difference between January
10th and January 11th is noticeably large. First, he will
notice that Jan. 10th is a Monday and Jan. 11th is a
Tuesday. Because this graph is a comprehensive
presentation of all divisions, he might want to look at
the behavior within each division on those particular
days. Graph 2 shows the % PTP Outbound calls within
one particular division, in this case, RCY_FSC. The %
PTPs on Jan. 10th and Jan. 11th are similar to the
relationship shown in Graph 2.
A division manager who is responsible for the
agents and teams within his division might wonder
what has caused the low performance on January 13 th.
The % PTPs for the RCY_FSC division for that day is
almost two standard deviations away from the mean
value.
P TP %
1.
2.
3.
4.
5.
2000 Systems Engineering Capstone Conference • University of Virginia
Graph 4. %PTP Outbound RCY_FSC
Agent AR1AAMIN
% PTP Outbound: RCY_FSC
40.00%
35.00%
%PTP Outbound RCY_FSC Agent AR1AAMIN
Date
% Calls Abandoned
Minus 1 Sigma
Minus 2 Sigma
Graph 2. % PTP Outbound RCY_FSC
In order to understand what has caused the pattern
of behavior in the above graph for the RCY_FSC
division, the division manager might talk to the team
leaders. The team leaders can each look at those
particular days on Graph 3 and describe what happened
on the team level.
% PTP Outbound RCY_FSC Team #1
25.00%
Percent
20.00%
Plus 1 Sigma
Plus 2 Sigma
Overall Average
SCORECARD SYSTEM REQUIREMENTS
There are two potential types of graphs on each
scorecard. One is the star chart, which displays
multiple metrics and the other are “up-down” graphs.
The “up-down” graphs display the individual data on
the x-axis and the percentage on the y-axis. Each
points is shown as a bar raised or dropped from the
mean value. Lines showing plus and minus one and
two standard deviations allow the reader to understand
the general variation of the data.
15.00%
Management Level
The management level scorecard includes
comprehensive data for the combined behavior of all
divisions according to various time series.
10.00%
5.00%
28-Jan
27-Jan
26-Jan
25-Jan
24-Jan
21-Jan
20-Jan
19-Jan
18-Jan
14-Jan
13-Jan
12-Jan
11-Jan
10-Jan
0.00%
Date
% Calls Abandoned
Plus 2 Sigma
Plus 1 Sigma
Minus 2 Sigma
Minus 1 Sigma
Overall Average
Graph 3. %PTP Outbound RCY_FSC Team #1
The agent level scorecard is used to notice agent
behavior over a period of time, rather than for one agent
on a particular day. First a particular agent might
perform on average below the norm. The team leader
would then look at that agent’s performance over a
period of time using Graph 4.
Division Level
The scorecard for each division is displayed
according to a time series. Because of the diverse
characteristics of each division, it is instructive to
observe data separated by division. These graphs are
shown in a time series format.
Team Level
The team level are similar to the agent level.
Various teams within one division can be displayed on
one graph to compare performance. Additionally, the
performance of one team across a time series can
provide additional information of the performance
specific to that team.
67
28-Jan
27-Jan
26-Jan
25-Jan
24-Jan
21-Jan
20-Jan
19-Jan
Minus 1 Sigma
Overall Average
18-Jan
Plus 1 Sigma
Minus 2 Sigma
14-Jan
% Outbound PTP
Plus 2 Sigma
13-Jan
Date
10-Jan
1/28/00
1/27/00
1/26/00
1/25/00
1/24/00
1/23/00
1/22/00
1/21/00
1/20/00
1/19/00
1/18/00
1/17/00
1/16/00
1/15/00
1/14/00
1/13/00
1/12/00
1/11/00
10.00%
1/10/00
15.00%
12-Jan
20.00%
45.00%
40.00%
35.00%
30.00%
25.00%
20.00%
15.00%
10.00%
5.00%
0.00%
11-Jan
25.00%
Percent
Percent
30.00%
Scorecard Development For A Debt Collection System
Agent Level
Agent level analysis should typically encompass a
time frame, which permits enough data to be gathered
for accurate statistical results. For example, one might
think that an agent with a PTP conversion rate of 90%
within the time frame of analysis is performing
extremely well. Statistically speaking however, that
conversion rate is not meaningful when the agent was
only passed ten RPCs and converted nine of them to
PTPs. Therefore, an agent’s total number of RPCs
must be taken into consideration when looking at the
PTP conversion rate for an agent.
The data used for the duration of this project
involved roughly three weeks of agent data. In most
cases, enough data was present to provide a significant
statistical analysis, but in some cases, more data would
be necessary. For this reason, agent analysis should
incorporate at least one month of data.
BIOGRAPHIES
CONCLUSIONS
John Congable is a fourth year Systems Engineering
major from Richmond, VA, concentrating in
Management Information Systems. When he is not
working on this Capstone Project, he can be found
beating his drums. Next year John will be working for
First Select Corporation in Pleasanton, CA.
The purpose of this Capstone project was to provide
First Select Corporation with an effective means for
performance evaluation. A scorecard for each level of
employment within FSC according to various time
series and employee series increases the value of
information contained in the graphs and tables of
performance. By consolidating this information for
comparison, management at FSC can use their time
efficiently.
REFERENCES
“Best Practices: Best Practices in Call Center
Management, Operations, and Technology”. ProSci
Benchmarking Report, 1999.
Carter, John. “Are you using the right metrics for
continuous improvement?” Production. Vol. 106,
Sept. 1994, 14-15.
Faulkner and Gray. “If the Call Center Fits…”
Business and Management Practices. Collections &
Credit Risk. July 1999, vol. 4 no. 7 p. 65-71.
Frost, Bob. “Performance Metrics: the new strategic
discipline.” Strategy & Leadership. Vol. 27, May-June
1999, p. 34-36.
Sprague, Gail. “Three Steps for Effective Call Center
Measures and Reports.” ACD Call Center: Online
Learning Center. Vanguard Communications.
68
Michael Autuoro is a fourth year Systems Engineering
major from Brooklyn, NY, concentrating in Economic
Systems. Mike has an incredible base of knowledge in
various New York sport teams and general computer
information; his wisdom is astounding. After
graduating, Mike plans on either entering one of the
distinguished law schools he has been accepted to or
entering the world as a professional in one of many
possible fields.
Hope Breskman is a fourth year Systems Engineering
major from Philadelphia, PA, concentrating in
Management Systems. Besides being incredibly active
in the Jewish community at UVA, Hope is also an avid
reader. This summer she will begin her financial career
with BlackRock Financial in New York City.
Arin Shapiro is a fourth year Systems Engineering
major from Corning, NY, concentrating in Economic
Systems. Besides dedicating his life to this Capstone
Project, he works at a local restaurant on the Corner and
encourages his teammates with a motivational cheer at
every meeting. Upon completion of the Systems
Engineering Curriculum, he plans to work as an IT
consultant for James Martin in Northern Virginia.
Rachel Yoo is a fourth year Systems Engineering major
from Atlanta, GA, concentrating in Biomedical
Management. When not participating in Honor
meetings, she hones her formatting skills. After
graduating, Rachel will be living in San Francisco, CA,
while working for A.T. Kearney as a Business Analyst.