Quali Huh?
Quali What?
Quali Who?
Qualimetrics
Combining Qualitative Approaches with
Quantitative Techniques to Heighten Strategic Impact
MRIA Learning Series October 17, 2005
Qualimetrics?
• Qualimetrics = Measurement or Metrics in a Qualitative
Research Project
• More accurate and positive descriptor than, say
Quali-Quant or Quanti-Qual, for area on the qualitative
side of the border between qualitative and quantitative
research
2
Qualimetrics Add Strategic Value
• Incorporating appropriate structured tasks/activities into
qualitative research design for focus groups, IDIs,
bulletin boards, theatre sessions:
– adds value to qualitative
– complements the classic open-ended discussion techniques
– allows for “processing” or analysis, using techniques
associated with quantitative
– the development of “rational graphics”, based on
qualimetrics, heightens impact/improves communication of
findings
3
Qualimetrics
• Looking for patterns and meaning:
– material generated in sessions
– “homework” assignments
– combination of secondary sources
• Many different approaches including:
– word counts
– coding of collages, images, videos
– processing worksheets
• As with all research design, qualimetric component(s)
should be driven by the specific objectives of the
research
4
B2B Project Examples
• From the Financial, Utility, and Medical sectors:
– Corporate image and repositioning
– Website evaluation
– Understanding attitudes towards the category and specific
products
– Optimizing incentive packages for renewal
5
Focus Group and IDI Approach
• Structured tasks and sorting of stimuli integrated into
focus group/interview guides
• Results processed and analyzed using techniques from
quantitative
• Development of “rational visuals” for presentation on
positioning and brand image
6
Typical Tasks Integrated in Guides
• Listing of businesses/products in the category/sources
of business information/websites visited/suggestions for
improvement in service/reasons for changing
suppliers/barriers for adopting technology
• Selection/sorting of images/taglines associated with
client and competition, and/or creating character
descriptions
• Circling/Marking words or phrases in text that have
value/resonate
• Completing worksheets with rating scales
7
Preparation of Stimuli/Worksheets
• For Qualimetrics, typically respondents work with
identical sets of images and stimuli
• Decks/sets of images/worksheets are prepared in
collaboration with client and agencies
• Stimuli may require lamination, standard size format,
and numbering/labeling on reverse to aid in coding
8
Processing of Imagery/Sorting Tasks
• Recording of the images/taglines/exhibits grouped or
selected for client’s brand and its competition using the
ID #
• Notes and verbatims of the character descriptions for
the images selected as most representative for each
brand
• Production of grids and hand tallies
• Preparation of materials for the project debrief, with
text summaries supported by images/tallies
9
Sample Reporting Slide on Brand Image
Fewer
associations
than competition:
–
–
–
–
–
–
–
–
traditional
conservative
rigid
old boy’s club
fraud
deceit
substance abuse
sexual deviancy
10
Processing of Circling/Marking Text Tasks
• Processing as with a quantitative survey:
– each sheet assigned a respondent ID number
– tabulation of number who circled individual words and
phrases
– may complete separate tabulations for total and subgroups
• Preparation of materials summarizing the results with
visual representation of the text with size of typeface
proportional to the number of respondents indicating
the word or phrase was valued:
larger typeface = higher value
smaller typeface = lower value
11
Sample Reporting Slide on Text
• Trust and support greatest value, especially to
employees
• Paragraph with “to encourage ourselves to be heroes”
generally inappropriate.
– How can you be a hero if you are doing what you’re supposed
to do?….Not a hero, just doing my job
• Employees also valued the following phrases more than
customers and non-customers:
– To stand together in the face of challenge
– To recognize each other’s contributions
12
Sample Reporting Slide on Text
All Respondents
Employees
together in the face of
challenge. To recognize each
other's contributions.
To stand
together in the face of challenge.
To recognize each other's contributions.
To
To stand
To
trust and support each other
so we might act in the best
other so we might act in the best interests
interests of our
of our customers.
customers.
encourage ourselves to be heroes, by providing
each other with the products and information our
To
To encourage ourselves to be heroes, by providing each
other with the products and
trust and support each
information our
customers really need.
customers really need.
13
Processing of Rating Sheets
• Processing as with a quantitative survey:
– each sheet assigned a respondent ID number
– tabulation with calculation of mean scores
– may prepare charts using multivariate approach, such as
mapping or dendograms
14
Sample Text from PA* in Focus Group Mode
●
●
●
New design was generally considered to enhance the corporate image as
more professional and efficiently run.
In the Perception Analyzer scores the new Website received average
ratings out of 10 of 7.4 for “fits in with your impression of the company”
and 7.1 for “enhances the company’s reputation”, compared to scores of
6.2 and a low 4.8 respectively for the current Website.
Typical spontaneous comments and notes about the new design include:
•
•
•
•
Very professional.
More hitech.
Conservative
Image projected is more
consistent (than current
site) with the company.
• Graphically it’s more corporate,
serious.
• The functionality of the site is very
corporate.
• (The image of the new site relative
to the current site) still in the same
building but it went through a
renovation.
* PERCEPTION ANALYZER AUDIENCE RESPONSE SYSTEM
Sample Chart from PA* in Focus Group Mode
WEB SITE ASSESSMENT
Average Mean Score Ratings out of 10
For Business, Consumer and Total Participants
New Site
Overall impression
Encourages browser to
explore
Ease of use
Relevance to you as
customer
Fits with your
impression of company
Enhances company’s
reputation
Current Site
Total
Bus.
Cons.
Total
Bus.
Cons.
(31)
(16)
(15)
(31)
(16)
(15)
6.9
6.4
7.5
5.5
5.3
5.7
6.8
7.7
6.5
7.9
7.1
7.5
5.2
6.4
4.9
6.1
5.5
6.7
6.1
6.1
6.0
5.8
5.6
6.1
7.4
7.5
7.3
6.2
5.9
6.5
7.1
6.9
7.4
4.8
4.4
5.2
* PERCEPTION ANALYZER AUDIENCE RESPONSE SYSTEM
Theatre Sessions with PA*/ARS/Key Pads
• Qualimetrics also covers measurements in large theatre
sessions using Perception Analyzer or similar
ARS/keypads. Typically these sessions can last up to an
hour and are followed by break-out focus groups.
• In addition to rating scales, you can conduct trade-offs
and moment-by-moment assessment of advertising and
videos.
• Results shown to clients during the sessions on screens
in viewing room. Data is processed as with a
quantitative survey.
* PERCEPTION ANALYZER AUDIENCE RESPONSE SYSTEM
17
Sample Charts From Theatre Sessions
•
•
•
•
Effectiveness and impact of incentive packages
Contact with client, and client follow-up call
Preferred partners for co-branding
Overall appeal of two direct mail packages
18
Effectiveness and Impact of Incentive Packages
Average Ratings of Four Packages
Scale of 1 to 100
#3
• 25% reduction in membership fees, or
• 2500 Bonus Points, or
• Special offers
58.8
68.4
#4
• Free cellular phone, or
• Long distance offer, or
• 2500 Bonus Points
58.1
46.7
#1
• Free membership rewards, or
• 2500 Bonus Points, or
• Long distance offer
41.9
47.6
#2
• Free Magazine subscription, or
• Free 1 year insurance registry, or
• One time free week of travel insurance
34.2
GROUP A
42.9
GROUP B
19
Communication of Cancellation
How Did You Notify Client of Cancellation?
Contact After Cancellation?
Don't Know
12%

72%

Returned
Product
Yes
52%
No
36%
16%
IF YES (n=13)
4%
8
Contact had no impact
Confirmation
12%
Contact made reconsider
4
Contact made reinstate
1
Reinforced desire to cancel
1
20
Preferred Partners for Co-branding
st
st
1
Choice
1 &
2nd
(A)
Least
Preferred
(B)
Adjusted
Score
(A-B)
Option #1
38%
65%
13%
52%
Option #2
27%
57%
18%
39%
Option #3
16%
43%
6%
37%
Option #4
13%
21%
25%
-4%
Option #5
7%
17%
39%
-22%
21
Overall Appeal of Two Mail Packages
Overall Appeal Score out of 100
“Informal”
Letter
60
“Formal”
Letter
60
Likelihood of Positive Response to the Letter
“Informal”
Letter
54%
23%
“Formal”
Letter
52%
11%
Very/Somewhat
Very
22
Challenges
• Additional effort preparing stimuli and processing task
notes and worksheets
• Budget impact as increases the executive commitment,
and/or technology if using PA/ARS/keypads
• Need to be consistent in terms of conducting the
tasks/exercises (versus flexibility in making changes to
guides/interview/sessions throughout the project)
• Qualimetrics lives in the borderline/grey areas between
qualitative and quantitative and, for some professionals,
is contentious
23
Conclusions
• Qualimetrics are a valuable and valid part of the
qualitative arsenal
• Qualimetrics do not force you into the strait-jacket
of quantitative research
• Relevant to B2B where target populations are smaller
e.g. IDIs with medical specialists
• Qualimetrics can liberate your creativity and add to the
value & actionability of the research
24