BUILDING A QUALITY ASSURANCE PROGRAM
FOR 2-1-1 CENTERS USING ‘VOICE OF THE
CUSTOMER’ RESEARCH
BARRY MANERS, MANAGING DIRECTOR, THE FRASER GROUP
LISA AUSTIN, DIRECTOR OF 2-1-1 STRATEGIC ENHANCEMENTS AND
DISASTER RECOVERY UNITED WAY WORLDWIDE
©The Fraser Group,2014
United
©The Fraser Group,2014
Way Worldwide
The Fraser Group
Voice of the Customer Solutions
RA
Road America
A MAPFRE Company
National 311 Performance Benchmark
amadeus
©The Fraser Group,2014
The Agenda

The 211 “mystery caller” benchmark program

Consensus reality, aka “Why does quality assurance
matter”

Lessons learned from the 211 “mystery caller”
benchmark program

The path forward
©The Fraser Group,2014
The Agenda

The 211 “mystery caller” benchmark
program
Consensus reality, aka “Why does quality assurance
matter”
 Lessons learned from the 211 “mystery caller” benchmark
program
 The path forward

©The Fraser Group,2014
The Agenda

The 211 “mystery caller” benchmark program
 Pilot
©The Fraser Group,2014
creation
How the 211 “Mystery
Caller” was created
Consultation




Determined Scope of Work
Worked with 211 US Steering Committee
Utilized AIRS Standards
Fraser Group Best Practices
 Florida
 Involved 211’s
©The Fraser Group,2014
Participating 2-1-1 Centers












1 - Broward County 2-1-1
2 - NJ Partnership 2-1-1
3 – United Way of Escambia County 2-1-1
4 – United Way of Northeast Louisiana 2-1-1
5 – WIN 2-1-1
6 – United Way of Greater Atlanta 2-1-1
7 – United Way of Greater Cleveland 2-1-1
8 - 2-1-1 LA County
9 – United Way Association of South Carolina
10 - NJ Partnership 2-1-1 (second location)
11 – 2-1-1 Palm Beach
12 – Michigan 2-1-1 (Gryphon Place)
©The Fraser Group,2014
The Agenda

The 211 “mystery caller” benchmark program
 Consensus
reality, aka “Why does
quality assurance matter”

Lessons learned from the 211 “mystery caller” benchmark
program

The path forward
©The Fraser Group,2014
Referral
/Loyalty
The
Customer
Caller
Perceived Value
Cost of
Acquisition
& Usage
Events
Assuring the caller has
the best experience.
Does it matter?
Overall
Quality of
Products &
Services
Promotions
The Contact
Center
Market
Presence
& Reputation
Support
Marketing
10
Understanding the
filter.
Referral
/Loyalty
Caller
Perceived Value
The Caller
Cost of
Acquisition
& Usage
Overall
Quality of
Products &
Services
Market
Presence
& Reputation
The Contact Center
The Filter
The Contact Center
Strategy
The
Organization
Goals/Objectives
Mission
Vision
11
Understanding the
filter.
Referral
/Loyalty
Caller
Perceived Value
Cost of
Acquisition
& Usage
The Caller
Overall
Quality of
Products &
Services
Market
Presence
& Reputation
The Contact Center
The Hawthorne Effect
The Contact Center
Strategy
The
Organization
Goals/Objectives
Mission
Vision
12
13
Referral
/Loyalty
Caller
Perceived Value
Cost of
Acquisition
& Usage
Overall
Quality of
Products &
Services
Market
Presence
& Reputation
The Contact Center
Frederick Taylor
The Contact Center
Strategy
Goals/Objectives
Mission
Vision
14
Referral
/Loyalty
Caller
Perceived Value
Key Performance
Indicators – Set 1
Cost of
Acquisition
& Usage
Overall
Quality of
Products &
Services
ASA
Market
Presence
& Reputation
Talk time
The Contact Center
Frederick Taylor
The Contact Center
Calls handled
per hour
Strategy
Goals/Objectives
Etc.
Mission
Vision
Abandon rate
Key Performance Indicators – Set 2
What
percent of calls are placed on hold properly by the SPECIALIST?
What percent of calls contained questions that were never answered by the
specialists, causing call backs?
What percent of calls contain hold patterns that cause abandons and call
backs?
What percent of calls would cause the caller to call back hoping for a different
specialist?
What percent of calls contain irrelevant conversation thereby increasing talk
time?
What percent of calls contain tragic phrases?
©The Fraser Group,2014
15
Difference between #1 and #2
– CONSENSUS REALITY
#
1
From the inside out, drives consensus
reality
#
2
From the outside in – the callers perception!
(the specifics of what the caller encounters
and the criteria the caller uses to evaluate
the center's performance)
©The Fraser Group,2014
16
The 211 Specialist Monitoring
Program Objectives
Performance
Improvement
Outcomes
Improved
Caller
Satisfaction
Reduced Cost
of Center
Operation
Patent Pending, 2007
Improved
Allocation
of
Technical
Resources
17
18
The 211 “Mystery Caller” Benchmark
Program
* Built on Fraser’s SETS Program (Service Excellence Tracking )
* Fraser’s model based on multicollinearity and regression
modeling of over 10,000 call center observations
* Refinement of the model, iterative between Fraser and 211
©The Fraser Group,2014
19
The 211 “Mystery Caller” Benchmark
Program
Fraser’s SETS Research Results:

5 major categories
The
Greeting (The Initial Contact)
Listening, Confirmation, Questioning (Assessment and Clarification)
Appropriate Solution (Information and Referral Giving)
Professionalism (Communication Techniques)
Closing (Closure)
Total of 50 specific criteria (results in a total of 2,500 observations for the
Benchmark)
 36 specific attributes of measurement
Augmented with 211 specific requirements developed qualitatively with
211
©The Fraser Group,2014
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial
Greeting and
Building Rapport
Closure
Assessment
and
Clarification
Communication
Techniques
Build the
Relationship with
Professionalism
Information and
Referral Giving
Patent Pending, 2007
20
How
was the pilot executed
©The Fraser Group,2014
Participating 2-1-1 Centers












1 - Broward County 2-1-1
2 - NJ Partnership 2-1-1
3 – United Way of Escambia County 2-1-1
4 – United Way of Northeast Louisiana 2-1-1
5 – WIN 2-1-1
6 – United Way of Greater Atlanta 2-1-1
7 – United Way of Greater Cleveland 2-1-1
8 - 2-1-1 LA County
9 – United Way Association of South Carolina
10 - NJ Partnership 2-1-1 (second location)
11 – 2-1-1 Palm Beach
12 – Michigan 2-1-1 (Gryphon Place)
©The Fraser Group,2014
The Fraser Assessment Scoring
Methodology

Each Attribute scored 0 or 1

Most attributes of call scored are objective, i.e.

Thank the caller for calling 211 of ________

Did the Specialist really listen to the caller’s
statements/questions?

Did the Specialist paraphrase the purpose of the call?

Did the Specialist handle the proper number of needs
as noted on the scenario used?

Did the Specialist avoid the use of jargon and tragic
phrases, unless explanation of the jargon was given?
©The Fraser Group,2014
23
Magic and Tragic Phrases
MAGIC Phrases (a partial list)
•I can help you with that right now
•I can look into that for you
•I can take care of that for you
•I’d be glad to help you
•I’m sure I can do that
•That would upset me too if that happened to me
•Mr. XXX. It would help me get your answer quicker if I
knew your order #, do you have that available?
•I’m glad I was able to help
24
25
Magic and Tragic Phrases
TRAGIC Phrases (a partial list)
•Please hold
•Give me your ZIP
•Hang on a minute
•What’s your problem?
•If you had read your confirmation more closely
•That would be too hard for us to administer
•Let me tell you what you haven’t considered
•You don’t understand our problem
•We have too many calls holding right now for that
•You have to
•Let me be honest with you
Outbound calls based on 23
scenarios covering
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Housing and Utilities-34%
Food and Meals – 26%
Information Services
Income Support
Individual& Family &Community support
Legal, Consumer &Public Safety
Health Care
Mental Health & Addictions
Clothing
Transportation – 2%
Other Govt./ Economic Services – 10%
Employment
Education
Disaster Services
Volunteers and Donations
Arts, Culture
Outbound calls based on
singular needs and unspoken
needs
SCENARIO
Caller:
43 YO
Male (or Female)
No children
Zip Code Needs – 2
Need type - 3
“Hi. I live in (CITY FROM 211 AGENCY BEING CALLED) and my electric is turned off. The Electric
Company isn’t being very nice because I can’t pay a bunch of money to get hooked back up. They
want $250 to reconnect and I can only pay about $100. I just lost my job and I just paid them $150 last
month and they still disconnected me. I think I owe $400 total, can you guys help me get my electricity
back on?
Outbound calls based on
singular needs and unspoken
needs
SCENARIO
Caller
Male or Female
26 YO
Lives in city that agency being called is in
Has had one leg amputated due to an auto accident
Zip Code Needs – 1
Need type - 12
I need help with moving. I have a lot of stuff and don’t have a truck and can’t afford to rent one. Is
there anything you can do for me?
Outbound calls based on
singular needs and unspoken
needs
SCENARIO
Caller
23 YO Female
Not married
2 children, 4 and 7
Lives in city that agency being called is in
Zip Code Needs – 2
Need type - 4
I need to find a food pantry to get some food to feed my kids but I can’t leave my
job during the day. Are there any that are open at night?
The Agenda

The 211 “mystery caller” benchmark program

Consensus reality, aka “Why does quality assurance matter”
 Lessons
learned from the 211 “mystery
caller” benchmark program

The path forward
©The Fraser Group,2014
Red Calls
• Specialist places caller on hold immediately upon connection
• Specialist uses profanity
• Specialist sets headset aside
• Specialist makes any disrespectful remark toward or laughs at the caller
• Specialist makes any disrespectful remark toward 211
• Specialist makes any disrespectful remark toward another agency
• Specialist talks to neighbor during the call
• Specialist asks the caller to call back (due to specialists or 211). If caller not prepared
with information, do not make a red call.
• Specialist is eating, smacking gum or has hard candy
©The Fraser Group,2014
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial Greeting and
Building Rapport
Closure
Assessment and
Clarification
Communication
Techniques
Build the Relationship with
Professionalism
Information and Referral
Giving
Patent Pending, 2007
32
Overall Calls Score
High Low Benchmark Total
100%
80%
60%
76%
76%
65%
50%
40%
63%
76%
67%
66%
52%
36%
48%
54%
High
Low
Total
20%
0%
-20%
0%
-40%
-60%
34%
36%
46%
-80%
37%
35%
Closing
Communication
Techniques
Information
and Referral
Giving
Assessment
and
Clarification
Contact
Overall Quality
-100%
Standard
Minimum
80%
33
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial
Greeting and
Building Rapport
Closure
Assessment and
Clarification
Communication
Techniques
Build the Relationship with
Professionalism
Information and Referral
Giving
Patent Pending, 2007
34
Contact
100% 100%
100%
100%
70%
Standard
Minimum
80%
80%
80%
80%
52%
60%
40%
27%
18%
20%
0%
-20%
0%
0%
0%
0%
15%
0%
-40%
-60%
-80%
-100%
Thank
caller for
calling
Agency
Identify
him/
herself by
name
Portray a
warm,
welcoming
tone
Personalize
call
Validate the
ZIP Code
35
High
Low
Total
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial
Greeting and
Building Rapport
Closure
Assessment
and
Clarification
Communication
Techniques
Build the Relationship with
Professionalism
Information and Referral
Giving
Patent Pending, 2007
36
Service Excellence Tracking
“Going Beyond Standard”
Assessment & Clarification
100%
100%
100%
87%
100%
95%
80%
80%
Standard
Minimum
80%
56%
60%
35%
40%
High
Low
Total
20%
0%
-20%
20%
80%
0%
0%
0%
0%
0%
-40%
-60%
-80%
-100%
Really
listen
No
inappropriate
interruptions
“I will
help” or
some
variation
Paraphrase
caller’s
question/
concern
Used open
and closed
end
questions
37
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial
Greeting and
Building Rapport
Closure
Assessment
and
Clarification
Communication
Techniques
Build the Relationship with
Professionalism
Information and
Referral Giving
Patent Pending, 2007
38
Service Excellence Tracking
“Going Beyond Standard”
Information Giving
100%
100%
100%
100%
83%
80%
Standard
Minimum
80%
75%
70%
60%
60%
40%
25%
20%
5%
0%
-20%
20%
60%
40%
40%
20%
Appear
knowl.
and
accurate
Handled
proper
number
of needs
0%
-40%
-60%
-80%
-100%
Response
clear and
understand
able
No
further
prompts
Did the AC
repeat
spelling of
addresses
/ vital info
39
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial
Greeting and
Building Rapport
Closure
Assessment
and
Clarification
Communication
Techniques
Build the
Relationship with
Professionalism
Information and
Referral Giving
Patent Pending, 2007
40
Service Excellence Tracking
“Going Beyond Standard”
Communication Techniques
100%
100%
100%
100%
100%
86%
80%
60%
58%
35%
40%
23%
15%
20%
18%
High
Low
Total
0
0%
-20%
Standard
Minimum
80%
80%
80%
20%
0%
0%
80%
0%
0
0%
Avoid
Jargon
Controlled
Silence
Gaps
Diffused
Anger
Expressed
Empathy
-40%
-60%
-80%
-100%
Ask
Avoid the
permission Apologize
inappropriate
to gain
for repeats
use of “we”
more
information
0 indicates criteria was NA in all calls
scored
41
Service Excellence Tracking
“Going Beyond Standard”
Communication Techniques
100%
100%
100%
100%
80%
80%
62%
100%
Standard
Minimum
80%
78%
68%
60%
50%
40%
27%
20%
0%
-20%
40%
20%
20%
60%
High
Low
Total
0%
-40%
-60%
-80%
-100%
Speak
clearly
Correct
pronunciation,
enunciation and
volume
Speak
courteously
Positive,
upbeat
tone
0 indicates criteria was NA in all calls
scored
Use the
caller’s
name twice
during the
call
42
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial
Greeting and
Building Rapport
Closure
Assessment
and
Clarification
Communication
Techniques
Build the
Relationship with
Professionalism
Information and
Referral Giving
Patent Pending, 2007
43
44
Closure
100%
80%
100%
100%
100%
Standard
Minimum
80%
68%
60%
0
indicates
criteria
was NA
in all
calls
scored
100% 95%
42%
40%
7%
20%
0%
-20%
20%
0%
0%
80%
Other
Issues?
Did the agent
avoid
Irrelevant
conversation
-40%
-60%
-80%
-100%
Attempt
to collect
demos.
Demos at
correct time
High
Low
Total
Service Excellence Tracking
“Going Beyond Standard”
Closing – Overall Centers Report
100%
100%
77%
80%
Standard
Minimum
80%
80%
60%
0
indicates
criteria
was NA
in all
calls
scored
40%
18%
20%
0%
-20%
60%
High
Low
Total
0%
-40%
-60%
-80%
-100%
Closer to a
resolution at
the end of
the call
Thank the
caller for
calling
agency or
211
45
Understanding the
filter.
Referral
/Loyalty
Caller
Perceived Value
The Caller
Cost of
Acquisition
& Usage
Overall
Quality of
Products &
Services
Market
Presence
& Reputation
The Contact Center
The Filter
The Contact Center
Strategy
The
Organization
Goals/Objectives
Mission
Vision
46
The 211 Specialist Monitoring
Program Objectives
Performance
Improvement
Outcomes
Improved
Caller
Satisfaction
Reduced Cost
of Center
Operation
Patent Pending, 2007
Improved
Allocation
of
Technical
Resources
47
Specialist Monitoring Using the Voice of
the Customer Through Mystery Calling
Overall Quality
of Specialist
Performance
Contact
The Initial
Greeting and
Building Rapport
Closure
Assessment
and
Clarification
Communication
Techniques
Build the
Relationship with
Professionalism
Information and
Referral Giving
Patent Pending, 2007
48
Overall Calls Score
High Low Benchmark Total
100%
80%
60%
76%
76%
65%
50%
40%
63%
76%
67%
66%
52%
36%
48%
54%
Standard
Minimum
80%
20%
High
Low
Total
0%
-20%
0%
-40%
-60%
34%
36%
46%
-80%
35%
Closing
Communication
Techniques
Information
and Referral
Giving
Assessment
and
Clarification
Contact
Overall Quality
-100%
37%
49
The Agenda

The 211 “mystery caller” benchmark program

Consensus reality, aka “Why does quality assurance
matter”

Lessons learned from the 211 “mystery caller” benchmark
program
 The
©The Fraser Group,2014
path forward
BUILDING A QUALITY ASSURANCE PROGRAM
FOR 2-1-1 CENTERS USING ‘VOICE OF THE
CUSTOMER’ RESEARCH
BARRY MANERS, MANAGING DIRECTOR, THE FRASER GROUP
LISA AUSTIN, DIRECTOR OF 2-1-1 STRATEGIC ENHANCEMENTS AND
DISASTER RECOVERY UNITED WAY WORLDWIDE
©The Fraser Group,2014