www.kdd.uncc.edu
CCI, UNC-Charlotte
Hierarchical Recommender System for Improving NPS.
presented by
Zbigniew W. Ras
Research sponsored by:
CONSULTING COMPANY in Charlotte
Client 1
Client 2
Client 3
shops
shops
shops
What we have to do
Client 4
Build
Recommender System
for each client
(34 clients) helping to
increase its revenue
Build Personalized
Recommender System
for each shop helping to
increase its revenue
Services
Parts
(heavy equipment repair)
CUSTOMERS
NPS of a company is correlated with its revenue growth
Net Promoter Score (NPS) –
today’s standard for measuring customer loyalty
Promoter - {9,10}
Passive – {8,7}
Detractor – {1,2,3…,6}
NPS rating for all clients
NPS rating for all shops
What is our goal?
Build recommender system giving possibly the simplest
and the best advice to every client/shop on the changes
they need to make in order to improve their NPS ratings.
Initial Dataset – provided by Consulting Company
About 100,000 records representing answers to a questionnaire
collected from over 35,000 randomly chosen customers in 2010 -2013.
Customers use services provided by 34 clients
A questionnaire consists of:
Information about the customer
Customer’s name, location, phone number…
Information about the service
Client’s name, invoice amount, service type…
Information on customers’ feeling about the service
was the job completed correctly
are you satisfied with the job
likelihood to refer to friends
…
Personal
Benchmarks
BenchmarkAllOverallSatisfaction
35
BenchmarkAllLikelihoodtobeRepeatCustomer
34
BenchmarkAllDealerCommunication
33
BenchmarkServiceRepairCompletedCorrectly
32
BenchmarkReferralBehavior
31
BenchmarkServiceFinalInvoiceMatchedExpectations
31
BenchmarkEaseofContact
30
BenchmarkAllDoesCustomerhaveFutureNeeds
28
BenchmarkServiceTechPromisedinExpectedTimeframe
26
BenchmarkServiceRepairCompletedWhenPromised
26
BenchmarkServiceTimelinessofInvoice
25
BenchmarkServiceAppointmentAvailability
24
BenchmarkServiceTechEquippedtodoJob
23
BenchmarkAllContactStatusofFutureNeeds
22
BenchmarkServiceTechArrivedWhenPromised
21
BenchmarkAllHasIssueBeenResolved
19
BenchmarkAllContactStatusofIssue
17
BenchmarkServiceTechnicianCommunication
6
BenchmarkServiceContactPreference
3
BenchmarkCBCallansweredpromptly
1
BenchmarkServiceReceivedQuoteforRepair
1
BenchmarkCBAutoattendantansweredbycorrectdepartment
1
BenchmarkServiceCallAnsweredQuickly
1
BenchmarkAllMarketingPermission
1
Randomly chosen customers are
asked to complete Questionnaire –
It has questions concerning
personal data + 30 benchmarks
To compute NPS we calculate average score
of all benchmarks for all customers. Knowing the
number of promoters and detractors we know NPS.
Classical Approach
Data
Data Preprocessing (including reduction of benchmarks)
Knowledge Extraction
(Classical Tools - WEKA +
Our Software for Extracting Action Rules & Their Triggers)
Recommender Systems
Classification
Selection of Best Classification Algorithm
0.4
0.405
0.41
0.415
0.42
0.425
0.43
0.435
Time Taken(secs)
Accuracy
PART
RBF
BayesNet
NaiveBayes
KNN
RandomForest
J48
0
10
20
30
40
50
60
70
80
PART
RBF
BayesNet
NaiveBayes
KNN
RandomForest
J48
Accuracy
2
3
4
1
5
7
6
Time Taken
2
4
6
7
3
1
5
Total score
4
7
10
8
8
8
11
11
Two Options
Initial Clients Datasets
2
1
Recommender Systems
local approach
global approach
Using Information about 34 Clients to Enlarge these Datasets
2
global approach
More Powerful Recommender Systems
FIRST APPROACH for Dataset Enlargement
Semantic Distance between Clients
J48 classifier extracted from
dataset D1 of Client-1
J48 classifier extracted from
Dataset for Client-2
dataset D2 of Client-2
Dataset for Client-1
attr1
attr2
Vector describing customer 1
Vector describing customer 1
attr2
attr3
attr4
attr3
attr2
Vector describing customer 2
Vector describe ing customer 3
attr1
attr3
attr4
attr3
attr1
Vector describing customer 2
Vector describe ing customer 3
More similar these two classification trees, more close semantically the clients are
Semantic distance based dendrogram
for Service
Recommender System Engine based on Semantic Similarity of Clients
Start with n1.
If NPS(n1) has to be improved, “move to n3”
If NPS(n2) > NPS(n1), then
Table3 := Table1  Table 2 is assigned to n3.
Classifier from Table3 is extracted and its
F-score is computed.
If F-score3 > F-score1,
then “move to n5”, otherwise STOP
n6
Table 5, Classifier5
n5
n7
n3
Table3
Classifier3
F-score3
n1
n2
\
NPS(n1)
Table1
Classifier1
F-score1
NPS(n2)
Table2
Classifier2
F-score2
For each client (node) we need to identify the best ancestor to be used for action rules discovery
Extracting Meta-Actions /sentiment analysis/:
We use Stanford Typed Dependencies Manual & Stanford Parser to generate grammatical relations
http://nlp.stanford.edu:8080/parser/
• Extract meta-actions from relevant comments that are associated with values of
attributes used in action rules;
• Find smallest sets of meta-actions triggering these action rules;
• Identify all customers supporting these action rules (needed to compute the
increase in revenue);
• Search for smallest groups of meta-actions that trigger maximal increase in
revenue (we present them in descending order for same-size groups of metaactions).
• effect(m) = num(distinct records in dataset involved with m) × conf(r*)
• Effect of a set of meta-action indicates the number of customers who are
expected to become promoters when these meta-actions are executed.
• Organize those groups of meta-actions hierarchically.
r1 = (A, a1  a2)  (B, b1->b2)  (C, c1  c2) 
(detractor  promoter)
Sup(r1) – number of customers supporting action rule r1 in a shop which NPS has to be improved
Number of detractors for a given shop having the properties (A,a1), (B,b1), (C,c1) - we target them
Before we start searching for triggers (meta-actions) supporting r1, this number is enlarged by adding customers
semantically similar and geographically close to our shop who also support rule r1.
Op1, Op2, Op3, Op4, Op5 – comments/sentiment about the service provided by this enlarged set of customers.
Comments about the service from promoters who satisfy the description (A,a2), (B,b2), (C,c2) are also listed.
Op1 - positive opinion related to A
Op2 - negative opinion related to C and B
Op3 – positive opinion related to B
Op4 – negative opinion related to A
Op5 – negative opinion related to C
Customers sentiment associated with rule r1 –
{Op1, Op2, Op3, Op4, Op5}
Trigger for A -> extracted from Op1 and not(Op4)
Trigger for B -> extracted from Op3 and not(Op2)
Trigger for C -> extracted from not(Op2) and not(Op5)
We search for minimum set of triggers covering A, B, C
Examples of Comments with Sentiment Orientation
staff+++++,best manager=112
staff-----,bad experience because diagnosis=108
staff+++++,good technician=96
staff+++++,nice guy=87
staff+++++,excellent mechanic=85
staff+++++,great guy=83
staff+++++,excellent technician=79
staff+++++,good guy=74
staff+++++,wonderful dealer=71
staff+++++,good team=66
staff+++++,honest guy=60
staff+++++,pleased with manager=40
staff-----,not available technician=34
staff-----,wrong diagnosis=16
staff+++++,best mechanic=12
staff+++++,knowledgeable mechanic=6
invoice+++++,outstanding bill=141
invoice+++++,they billed properly=65
invoice+++++,fine invoice=61
invoice-----,refused pay bills=29
invoice+++++,happy with bill=12
price-----,not fair pricing=108
price+++++,good price=108
price+++++,fair pricing=87
price-----,aggressive pricing=66
price-----,unreasonable charge fee=37
price-----,not satisfied with price=35
price+++++,better pricing=27
price-----,expensive amount charged=12
price+++++,the charged fairly=12
How to construct optimal sets of meta-actions (triggers activating
changes in customer NPS)
Threshold for adding triggers = 6
Seeds
We add single trigger
one by one following
their order with respect
to support
Sup=180
{TR5,TR1}
Sup=160
{TR5}
{TR1,TR4}
Sup=165
Sup=163
{TR5, TR2}
TR5, TR3}
Triggers activating changes in customer NPS
Sup=184
Sup=189
Sup=185
{TR5,TR1,TR2}
{TR5,TR1,TR3}
{TR5,TR1,TR4}
1
2
3
TR5
{TR1,TR4}
{TR5,TR1,TR3}
{TR5,TR1}
4
Search for the best meta-actions
• Client-2 is selected to be our target.
Client{2,4,6,1} by HAMIS
NPS
0.77 (73 detractors)  0.852
Action rules extracted
28,126
Meta-actions extracted
15
Number of meta-nodes
447
(*But we only focus on the top 10 in
each size category of meta-nodes.)
Maximum effect in meta-nodes
45.83 (out of 73 detractors)
24
Search for the best sets of meta-actions
for Client-2
idx
Meta-actions
effect
0
• ensure service done correctly
2.00
1
• ensure service done correctly
• improve price competitiveness
7.62
2
• ensure service done correctly
• properly set invoice
expectations(slightly high)
5.0
3
• ensure service done correctly
• sufficient staff
2.0
7
• ensure service done correctly
• competitive price
• improve price competitiveness
11.13
8
• ensure service done correctly
• improve price competitiveness
• sufficient staff
10.66
9
• ensure service done correctly
• improve price competitiveness
• properly set invoice
expectations(slightly high)
9.99
• ensure service done correctly
• improve price competitiveness
• keep proactive communication
8.62
10
idx
Meta-actions
effect
22
• ensure service done correctly
• competitive price
• improve price competitiveness
• sufficient staff
15.07
23
• ensure service done correctly
• improve price competitiveness
• properly set invoice
expectations(slightly high)
• sufficient staff
15.00
24
• ensure service done correctly
• competitive price
• improve price competitiveness
• reasonable invoice
13.13
25
• ensure service done correctly
• improve price competitiveness
• reasonable invoice
• sufficient staff
11.66
46
• ensure service done correctly
• competitive price
• improve price competitiveness
• properly set invoice
expectations(slightly high)
• sufficient staff
19.41
Search for the best meta-actions
• Why HAMIS helps?
1. 2) Extra meta-actions extracted from datasets extended by HAMIS help
improve effect further.
Set of meta-actions
Expected effect
competitive price
ensure service done correctly
improve price competitiveness
sufficient staff
9.87
decrease dealer response time*
11.13
competitive price
ensure service done correctly
improve price competitiveness
keep proactive communication
nice technician
reasonable invoice
7.59
knowledgeable technician*
13.3
26
QUESTIONS?