On The Qualitative Frontline:
Fieldwork Recruiters’ Views On
Respondent Cooperation
Prepared by:
QRCA Field Committee
March 2010
1
Table of Contents
Study Overview
Background
Study Objectives
Methodology
Note of Thanks
Caveat
Summary and Implications
Detailed Discussion
Trends in Respondent Cooperation
Factors that Affect Respondent Cooperation
Recruiter Suggestions for Decreasing Cheating/Repeating
Recruiter Suggestions for Increasing Respondent Cooperation
Appendix A: Moderator’s Guide
3
4
5
6
8
9
10
26
27
30
56
61
73
Page hyperlinks work directly in the Slide Presentation View, otherwise right click on page links above,
then click ‘open hyperlink’ to jump directly to a page
2
STUDY OVERVIEW
3
Background
•
QRCA’s Field Committee formed a Task Force to gain insights into respondent
cooperation in qualitative market research studies by interviewing fieldwork
service recruiters. The Committee believes that recruiters are “on the frontline”
and, therefore, can be a valuable source of information and insights.
•
The goal is to understand what qualitative research consultants (QRCs), fieldwork
services (facilities), and clients can do to increase respondent cooperation, by:
– Developing best practices for increasing respondent cooperation rates.
– Improving communication between QRCs/clients and fieldwork services on recruiting
practices.
– Reducing the incidence of cheating and repeating (respondents who lie in order to get
into studies and/or to participate too often).
4
Study Objectives
•
Specific study objectives are to:
– Identify trends in respondent cooperation
– Identify factors that increase or decrease respondent cooperation
– Obtain recruiters’ suggestions for increasing respondent cooperation
– Obtain recruiters’ suggestions for reducing the incidence of cheating/
repeating
5
Methodology
•
Five QRCA Field Committee members who are practicing qualitative research
consultants, most of them also facility owners, volunteered for the Task Force.
Each conducted 2 groups with recruiters.
– Jan Lohs, Lohs Research Group, in association with Smith Research, Chicago, IL
– Anndel Martin, Opinions Unlimited, Houston/Dallas, TX
– Andrea Schrager, Meadowlands and New York Consumer Centers, New Jersey and New
York
– Merrill Shugoll, Shugoll Research, Bethesda, MD
– Diane Trotta, Trotta Associates, Los Angeles and Irvine, CA
6
Methodology
•
A series of 10 focus groups was conducted, divided between:
– Recruiters specializing in consumer recruiting
– Recruiters specializing in executive/professional and medical recruiting
•
Focus groups were about 90 minutes, with 6-8 respondents each.
•
Recruiters interviewed were a mix in terms of recruiting experience (minimum 6
months), gender and age.
•
Recruitment criteria and the moderator’s topic guide, developed by the Task Force,
are appended.
7
Note of Thanks
•
Special thanks to the following companies for providing transcription services:
Transcription Services, Inc
TSItranscripts.com
909-484-8185
Mark Hampton Transcripts
markhampton88@yahoo.com
212-595-4083
MCC Field and Transcription
Nadya@consumercenters.com
201-537-5256
8
Caveat
•
The qualitative research methodology seeks to develop direction rather than
quantitatively precise measures. Because of the limited number of respondents
involved in this type of research, the study should be regarded as exploratory in
nature and the results should be used to generate hypotheses for further testing.
•
This study presents views of qualitative recruiters. Comments by the Field
Committee are discussed separately.
9
SUMMARY AND
IMPLICATIONS
10
A. RECRUITERS’ REPORTS
AND SUGGESTIONS
11
Trends in Participation
•
Several trends are contributing to respondent participation
– Awareness – more people know about market research generally and qualitative in
particular, especially focus groups.
– Positive past participation – people who have participated usually have a good
experience, are more likely to come again, give referrals, spread word-of-mouth.
– Recession – increases interest because of incentives.
12
Trends in Participation
•
But certain trends present obstacles to respondent cooperation:
– Continuing lack of research awareness among some consumers and office gatekeepers.
– Company policies against employee participation, affecting business-to-business studies.
– The recession – more people working longer hours, don’t want to leave office for a
study.
– Busy lifestyles – less time for screening, participation.
– Cellphone use – harder to reach respondents, safety issues while driving.
13
Factors in Cooperation: Overview
•
Recruiters/facilities look at cooperation both in terms of the individual study and
keeping respondents in their database long-term.
•
The screener plays a major role in cooperation (discussed below).
14
Factors Helping Cooperation
•
Study topic – those respondents consider interesting, personally relevant.
•
Incentives – appropriate to the market, respondent segment, amount of pre-work.
•
Facility contact prior to screening – emails/letters/fax introducing the research
company are a good first step in recruiting.
•
The actual research experience – respondents who find the interview worthwhile/
enjoyable are open to participating again; seeing results of the research makes
them feel they have had an impact (on product development, etc.).
15
Factors Hurting Cooperation:
Sampling/Requirements
•
Narrow study requirements, quotas – database respondents become frustrated to
be called repeatedly but not qualify; some opt out.
•
Client list studies – inaccurate information is common; respondents are wary if not
told the client name.
•
Random digit dialing – people unaware of research are less likely to participate.
16
Factors Hurting Cooperation:
Sampling/Requirements
•
Homework assignments have become more demanding – resulting in lower
acceptance and show rates.
•
Incentives – too low for homework required, no incentive for completed
homework if respondents are cancelled; respondents refuse if less than prior
studies.
•
Scheduling – times of day that are difficult for the respondent segment.
•
Last-minute changes – in interview scheduling, new/revised screener questions.
•
Confirmation of holds – if not done quickly, respondents make other plans.
17
Factors Hurting Cooperation:
Screeners
• Length – many screeners are longer today; with questions “for information” or as a
mini-survey, rather than for the purpose of screening.
• Questions that are complicated or poorly written are more common:
– Unclear wording.
– Run-on sentences.
– Long response lists.
– Respondents can’t remember what they did a while ago, especially everyday usage.
– Redundant questions, repeated verbal scales annoy/offend respondents.
– Terminology that is hard to pronounce/understand.
– Poorly translated from other languages.
• Organization/flow
– Difficult skip patterns
– Terminate questions at end – waste respondents’ and recruiters’ time.
18
Cheating/Repeating
•
Cheaters/repeaters have become more knowledgeable about “gaming the system”
in order to get into studies.
– The recession raises facility concerns about respondents doing this for the money, lying
about employment.
•
Facility procedures can help reduce cheaters/repeaters.
– Electronic monitoring recruiting, confirmation calls checking screener information using a
different recruiter.
– Keeping the database updated on participation.
19
Recruiters’ Suggestions: Screeners
•
•
•
•
•
•
•
Introduction – be engaging and honest about the study topic/purpose; ask the
project manager how to present the subject in an interesting way.
Respondents desired – talk with the project manager about what you want.
List studies – tell respondents the client name, if possible.
Length – keep screeners short (under 10 minutes at most), ask only questions
needed to recruit.
Flow – make skip patterns and instructions simple/clear.
Terminates – do this on the question after the one where respondents disqualify,
especially on sensitive issues (e.g., age, ethnicity)
Algorithms – use them only if absolutely necessary; recognize they complicate
recruiting and increase costs.
20
Recruiter Suggestions: Screener
Questions
•
•
•
•
•
•
•
Ask key questions, including security, in an open-ended way, without reading
pre-coded answers – more difficult for cheaters/repeaters to game the system.
Ask children questions in simple, age-appropriate language.
Use numeric scales instead of verbal ones on a list of statements.
Ask articulation questions related to study subject – avoid ones seen as “silly.”
Put terminate questions as close to beginning as possible.
Phonetically spell out difficult terminology.
Translate screeners originally in another language into colloquial English.
21
Recruiter Suggestions: Other Issues
•
•
•
•
•
•
Requirements/quotas – recognize that studies with very narrow specifications are
more difficult, expensive, take longer to recruit; ethnicity is more complex today.
Incentives – consult with the project manager on what’s appropriate to the
market, segment, amount of pre-work required; partial payment for cancelled
respondents who complete homework; give a small gift in addition to cash at the
end of the interview to create goodwill.
Confirm holds – do this as soon as possible to retain desired respondents.
Homework – keep it to a minimum, especially for businesspeople; make help
available on weekends for Web assignments; consider cameras as incentives.
Scheduling – ask the project manager for the best interview start times in that
market; consider weekends and holidays for longer interviews (e.g., mock juries) or
to recruit certain segments (e.g., students).
Refreshments – serve meals at standard mealtimes.
22
A. FIELD COMMITTEE
COMMENTS
23
Overview
•
This study with qualitative research recruiters provides a number of very useful
observations and suggestions for QRCs and end clients in planning their fieldwork
in today’s world.
– QRCs should try to avoid practices that cause recruiting problems and/or price increases.
– QRCs should make their clients aware of the consequences of practices that complicate
recruiting and/or are more costly.
•
•
It is important that QRCs/clients work collaboratively with facility project
managers, asking their advice on recruiting and their particular market.
We also strongly recommend that QRCs and facilities follow the guidelines for best
practices in communication and recruiting practices, including email recruitment,
on the QRCA website.
24
Better Communication
•
There are reasons why QRCs/clients do not or cannot follow practices suggested by
recruiters. Explaining the reasons to the facility helps the project manager and
recruiters understand the situation. Examples:
– Study specifications may be narrow in order to find people relevant to the client.
– Homework assignments sometimes need to have multiple parts in order to enrich the
research.
– Modifications in respondent qualifications may be necessary after interviews in one
market, such as learning that certain types of people are not true prospects for the
client.
– Small changes in an answer (e.g., one rating point) may disqualify a respondent – while
this is frustrating, it may be necessary.
– Respondent callbacks to ask additional or revised questions may be needed based on
new information/ideas or on the mix of recruited respondents.
– A few terminate questions may need to be toward the end of the screener – e.g.,
respondents can be put off if household income is asked too early.
25
B. DETAILED DISCUSSION
26
1: TRENDS IN RESPONDENT
COOPERATION
27
Respondent Cooperation Trends
•
The economic downturn is having a slight positive impact on respondent
cooperation. More people are returning recruiters’ calls and agreeing to
participate in qualitative research.
“Maybe it has to do with the economy; everyone needs more money.”
“People are more willing to listen to you and take your call. They will put dinner on hold
to answer your questions – the kids could be screaming in the background but they keep
on going.”
•
Negatively, business-to-business studies can be more difficult to recruit today.
– Some professionals have heavier-than-usual workloads and are concerned about being
out of the office during the recession.
– More companies have policies against employees’ participation in market research.
28
Respondent Cooperation Trends
•
Many people remain unfamiliar with market research. They are more likely to
disengage from a phone conversation with a recruiter, are skeptical when told that
the research company wants their opinions and will compensate them for their
time.
“Executive and medical recruiting is more difficult if the gatekeeper is not familiar with
qualitative research.”
“They have to be familiar in order to trust you with whatever you are talking about.
When you call them, they do not know who you are. You mention money – ‘we’re going
to pay you’ – that is a scam in their mind.”
“If you are not allowed to say the company that you are calling on behalf of and they
don’t know anything about you, they don’t care; they don’t want to talk to you because
they don’t know anything about a focus group. They don’t know about marketing
research.”
“’Are you serious! You pay me?’”
“You can’t get past the administrator. They think you are trying to sell something; they
are paid to protect the doctor, not let people get through to them.”
29
2: FACTORS IN RESPONDENT
COOPERATION
30
Respondents’ Past Experience
•
Many respondents have had positive experiences in qualitative studies, which
makes them willing to participate again.
– They enjoyed expressing their opinions and being heard. Most are anxious the first time
and then find the experience pleasurable.
– Seeing an actual outcome of the research (e.g., new product/advertising discussed
come to market) makes respondents excited about having had an impact.
•
Respondents expect the same level of incentives in future studies, are less likely to
participate when it is lower.
“There are people who have told me [no] if it’s not $100.”
31
Study Topic
•
Interesting or personally relevant topics attract people to participate. The gratuity
becomes secondary, although it is still important.
“I think it depends on the topic. It could be something that’s very interesting to them as a
person and that makes them much more interested in participating.”
“They will make an effort to come, even if it’s inconvenient, because they enjoy it [the
topic].”
“Some doctors want to learn about new techniques. They’re not just doing it for the
money. And they want to get feedback about medications from other doctors.”
•
Topics that may not sound inherently interesting can be positioned in a positive
(and honest) way.
–
Example: a study on “what consumers do for personal grooming” vs. “a study of
deodorants.”
32
Study Topic
•
If the actual interview topic differs from what respondents expected/were told or
is about issues of little interest to them, they may feel misled or be reluctant to
participate again.
“Very often, we find out after the fact that the vague topic ended up being an ad
campaign. ‘What color or what font should we be using?’ Some doctors can get angry
to think they have attended something that was quite different than what they thought
it would be.”
33
Screener Introduction
•
Introductions need to capture respondent interest, provide reassuring information.
–
Generic introductions that give no real information about the study topic (e.g., “some
issues”) do not intrigue respondents.
–
Clients willing to reveal their sponsorship of the study, especially on a list job, sometimes
leave it out of the introduction, which can result in terminates by people who would be
interested in participating.
–
The recruiter’s tone at the outset also plays a role in encouraging participation.
“I think a good intro is one where we can say within a couple of sentences who we are,
what we’re trying to do, and if you qualify we’d like to invite you to participate.”
“The more the qualitative research people can provide you with ‘this is what’s going to
be happening in the group,’ the more we can really sell them on the idea of coming to
the session.”
“Sometimes it’s not what you say but how you say it. If the topic sounds interesting in
your introduction I think you can sell the idea that it will be fun to participate.”
34
Screener Length
•
Screeners have been growing longer (up to 20 minutes), creating major problems
with participation rates.
– More screeners today include questions for general information purposes or as a minisurvey, which goes beyond screening.
– Screeners longer than about 7 minutes can cause respondents to become frustrated, to
lose interest and terminate midway.
“Some questions are a whole paragraph, and that’s not necessary. By the time you get
to the question, they’ve already forgotten.”
“Respondents sometimes feel like the screener goes on forever and they get lost in the
detail.”
•
Terminate questions near the end of a screener annoy respondents, waste both
their and the recruiters’ time.
“I try to get permission from the project manager to go directly to the termination
question [when I begin a screener].”
35
Screener Questions
•
Simple and direct questions are most easy to answer correctly.
– Long questions with run-on sentences confuse respondents – they don’t know what part
of the sentence they should answer.
– A difficult-to-administer screener makes respondents think the study will also be
difficult.
“Respondents tune out to long lists. They just answer and are not listening to what you
are really saying.”
“Directions can be so long that [respondents] don’t remember what they are doing.”
36
Screener Questions
•
Numeric scales with anchor points are simpler, less confusing and more timeefficient than verbal scales that recruiters have to reread for each statement.
– Numeric scale example: “for each statement, choose a number from 1 to 5, where 1
means disagree completely and 5 means agree completely.” Verbal scale example: “for
each statement, tell me if you agree completely, agree somewhat, neither agree nor
disagree, disagree somewhat, disagree completely.”
“You have to make sure that they understand the scale. Sometimes when they are filling
out a rescreener before the group, they answer differently and I think it is because they
didn’t understand the scale on the phone because it wasn’t logical or explained well.”
“When I ask how many times they were at a particular restaurant or drank a beverage
in the last year and then drill down to the last month, the numbers change and the
respondent gets confused. Start with the last month. That changes when you ask about
purchasing a big item like a car or taking a cruise. Those things they can remember.”
37
Screener Questions
•
Redundant questions are seen as being used to verify answers – they can annoy or
even offend some respondents, who feel they are being tricked.
“They say, ‘Do not waste my time.’”
•
Skip patterns are often not clear – more screeners now are questionnaires with
complicated, hard-to-follow instructions.
“The screeners used to be simpler and they have become more complicated over time.
It’s not about ‘keep it simple.’”
•
Some screeners originally in foreign languages are not translated well.
“With doctors, they use the same screener in Europe, India, China, wherever, and
sometimes it doesn’t fit into our language or how we talk.”
•
Terminology that is not commonplace lay language (e.g., financial, medical) can be
difficult for recruiters to pronounce if they are not spelled out phonetically.
“I had a couple of screeners where it seemed like the screeners were like research
papers. Some of them are cumbersome; they’re poorly written. The terminology they
use do not encourage a successful completion.”
38
Study Specifications
•
Increasingly, narrow specifications and quotas mean that fewer people qualify for
study participation.
– Respondents in facility databases become discouraged about repeatedly failing to
qualify, and are negative about being screened for other studies.
“I’ve had some get very angry on the phone. They say, ‘Why are you wasting my time? I
never qualify.’”
39
Algorithms
•
Algorithms are a major concern – these multi-question formulas have to be
calculated, recruiters cannot determine immediately if a respondent qualifies.
– Respondents feel their time has been wasted if they make it through most of the
screener, then don’t qualify because of the algorithm. They may be less willing to
be screened for the future studies.
“The problem with the algorithm is that the person doing the interview can’t
determine if they qualify.”
“Respondents do not want to waste a lot more time on the phone while I figure out
if they qualify or not. They get annoyed and hang up if the calculation takes too
long.”
“It builds frustration. If it happens too often, they just say ‘I don’t want to do this
any more.’”
– Some recruiters explain the rationale to respondents (clients look for people based
on their attitudes); calculate the algorithm offline, calling back those who qualify.
“Once I understood what an algorithm was and why it is important, I could help the
respondent feel more comfortable going through the process.”
– Some facilities pre-screen respondents with a form online, screen those who pass.
40
Follow-Up Calls
•
Last-minute changes in interview scheduling, screener questions and respondent
approval are more common today – these irritate and put off respondents.
– Calling respondents back with new questions is seen as an imposition. A small change in
an answer (e.g., one rating point) can result in disqualification.
“Once we are halfway through with the recruit, they’ll [the client] call back with a new
question.”
“You feel badly when you have to go through lots of questions that they already
answered.”
“It is ridiculous sometimes. Once they give a 7 and then an 8 when re-asked and now
they do not qualify.”
41
Confirming Holds
•
Respondents put “on hold” for participation because they do not totally fit
screening qualifications can become annoyed or make other plans if the client
“waits too long” in accepting them for the interview.
“Sometimes, the client won’t get back to the project manager until a day or two before
[the session]. We’ve lost holds because they’re not available anymore.”
42
Incentives
•
Respondent cooperation is largely affected by the amount and type of gratuity.
– The right amount varies by market – the project manager has experience to know what
works best in their market with different segments.
– Beyond the basic incentive, it can be useful to pay additional gratuities, such as for
completed homework, for parking, and drawings for early arrival at the facility.
– While high-interest topics increase participation rates, this does not mean that lower
incentives are a good idea – show rates can be hurt.
“If it’s two hours, and it’s $75, they’re usually not interested. I would say if you have
somewhere between $100 to $125, you’ve got them.”
“Some people, that’s the first thing they’ll ask me, ‘Is the gratuity [amount]?’”
“I don’t know why but respondents seem to like it better when we separate the incentive
for babysitting or parking rather than telling them one lump sum amount.”
43
Sampling Method
•
Cold calling through random digit dialing is the most challenging method – many
people contacted may not be familiar with research generally or the company.
– Similarly, client list studies on which recruiters are not authorized to name the study
sponsor are difficult – respondents are wary about where the research company got
their name/information.
“[They] are more suspicious, they don’t know who we are, what we do. You have to give
them a level of comfort, or they’ll hang up on you.”
“’I’m so-and-so from such-and-such’ – it suddenly sounds like a telemarketing call.”
“We need to spend a lot of time explaining the nature of the call, the difference between
telemarketing and research to establish their trust.”
“A lot of the time they think I’m a bill collector until I tell them about [our company].”
44
Sampling Method
•
Client lists need to be up-to-date, with full contact information. Many lists have
inaccuracies in addresses, telephone numbers and respondent information.
– Recruiters sometimes are not warned that people listed may have negative reactions to
a call and/or to the client.
“There are a lot of wrong numbers. It seems like some lists are old.”
"It is great when lists have more than one phone number. You can try them during the
day or at night.”
“We need to know the type of person on the list so we can calm them down. If they are
people who have called to complain, someone with a lousy credit history, they can get
angry.”
45
Contacts Before Calls
•
Emails, letters by mail or a fax introducing the research company and the study can
be a helpful first step in recruiting.
– With respondents who know the facility, these steps avoid calls bothering them about a
study they are not qualified for or interested in.
“It is best that way. They want to be included if they respond to an email.”
– People on a client list have heard the research company’s name when they are called,
making them more receptive.
“If we can, emailing ahead of time really helps. They know the name [of the research
company] and they will answer my call.”
46
B2B and Consumers
•
Recruiting B2B respondents is generally more challenging than consumers.
– Businesspeople and professionals are often more difficult to reach, have less time to be
screened or to participate in a study.
“For business recruits, it’s more challenging because you’re interfering in their day,
because it’s hard to get them on the phone to screen them, and then it’s hard to get
them to come in. It’s easier with consumers because word-of-mouth and past
participation makes them want to do it again.”
“You need to get through, you have to work your way through the people, the
receptionist.”
•
Consumers are easier to recruit than businesspeople, once they are contacted,
because of experiences in previous studies or word-of-mouth recommendations.
– Characteristics like age and gender do not seem to make a major difference in
respondent cooperation but do play a role in show rates.
“It’s not that younger/older is easier or harder to recruit, it depends on whether the topic
relates to them or not.”
47
Type of Interview
•
Most respondents are willing to participate in a range of interview types.
– Businesspeople /professionals have divided preferences for depth interviews and focus
groups. Depth interviews tend to be shorter and have greater flexibility in scheduling,
giving respondents choice of the time. Focus groups give them the opportunity to hear
others’ opinions and learn from the interaction.
“Sometimes they say they only want to do groups – especially if the topic is interesting
or they get to learn about new products from others.”
•
Respondent concerns about ethnography home visits were noted by some
recruiters.
– Even respondents who know/trust the field service can be hesitant about strangers
visiting; researchers coming to see how they live seems foreign/“weird” to some.
– Respondents want to know details, e.g., who will be visiting, the visit purpose. Securing
cooperation is especially difficult when recruiters cannot tell them. Some women will
not allow males into their home unless their husband or a friend is present.
“They’re like, ‘You want to come in and watch my family eat dinner? I don’t feel
comfortable with that.’”
“It is the fear factor. They know my company but do not know the person coming. They
sometimes ask if I can be there too.”
“They are just uncomfortable and it is even worse if more than one person is going. They
feel overpowered in their space.”
48
Type of Interview
•
Store shopalongs are easier to recruit than home visits but some respondents
express concerns that they might be asked to make purchases.
“I get asked these days if I have to purchase things. They say they just shopped and do
not want to spend more money on things they do not need at the moment.“
49
Time of Day
•
Interviews scheduled at times when respondents are busy/unavailable make
recruiting far more difficult.
“In the old days, it was what was convenient for the respondent and now it’s what’s
convenient for the researcher or the clients.”
•
Best start times for early evening interviews vary by market.
– Traffic patterns and distance to the facility from the highway or downtown area need
to be considered.
“If a focus group is scheduled any time around rush hour, it is a problem for some to
come.”
“For respondents who live or work far away from the facility, getting to a 6pm group can
be very challenging during rush hour traffic.”
50
Time of Day
•
Demographic segments have different time preferences that should be considered;
these also vary by market.
– Elderly respondents generally prefer daytime or early evening to 8pm interviews.
– Businesspeople and professionals generally prefer early morning and evening for focus
groups. Midday can work for shorter IDIs but not for two-hour focus groups.
“Maybe a year ago doctors were willing to reschedule a patient. Now they are not as
willing to tweak their schedule in fear of losing a patient.”
“We find that many doctors do not see patients on Wednesdays or Fridays so those days
work well.”
•
Serving a meal near/around typical mealtimes encourages cooperation.
“For a 4:00 pm group we are asked, ‘You are serving dinner, aren’t you?’”
51
Day of the Week
•
Certain days of the week can be a problem in some communities (e.g., church
night) – the project manager can advise on this.
•
Weekends and school holidays may be preferred by some consumers, especially
for longer interviews and mock trials – their schedules may more flexible then.
– However, some respondents, researchers and facilities do not like these times.
52
Pre-Interview Homework
•
“Homework” assignments are increasingly used today. While a number of
respondents find these interesting, they can make recruiting more difficult and
require more recruiting time.
– Certain professionals dislike or won’t do homework assignments, according to some
recruiters, while other recruiters see no problem.
“The doctors don’t like doing homework at all. They did enough of that in school.”
“I actually heard a doctor ask his nurse to pull ads from magazines for him for his homework when
called to remind him to bring it in. I tried to stop him but do not know who actually did it.”
“I don’t think they mind doing the homework. They’ll laugh about it, but for the most part, they’re all
right with it.”
– “One-step” homework is easier to “sell” to respondents than multiple tasks.
“If we have jobs with homework, we should allow at least one weekend because they don’t have a lot
of time for themselves. If we have a turnaround of a week, then they don’t have time to do it.”
“I think they (respondents) like to do homework sometimes if it is fun and not too much work. A 50page diary is way too much work; [sometimes] they don’t show up because they didn’t do it.”
53
Pre-Interview Homework
•
Homework that is not clearly spelled out, is more difficult or takes more
time/effort than promised may lead to respondent frustration, cancellations and
no-shows.
– Similarly, incentives that are seen as insufficient for the assignment are a problem.
“[On a recent project] respondents were angry. They said ‘even though you told me 3 to
4 hours, it’s taking me 5 or 6.’”
“I have had people drop out of the study because the homework is much more involved
than either I or the respondent expected.”
•
Incentives that increase cooperation include a still or video camera for doing the
homework, a bonus for the most complete homework.
•
Cancelled respondents who complete and submit homework become upset, are
less likely to cooperate on future studies.
– Offering partial incentives in these cases promotes cooperation.
54
Cancellations
•
Last-minute cancellations create negative feelings and affect future cooperation.
– Client changes in study specifications near the interview date can result in the
cancellation of some respondents. This is often done without explanation or
compensation. Respondents who adjusted their schedules to be available resent the
short notice.
“The client changes the specifications and tells us to replace respondents. The client may
have numerous reasons why and they don’t want to tell the respondents why we can’t
use them.”
“That leaves a bad taste in their mouths. The next time we call to recruit them, they’re
going to be skeptical and not take us seriously. They need advance notice if they’re going
to be cancelled.”
“We feel that these people should get paid if cancelled at the last minute. Often they
have rescheduled other meetings around our schedule.”
55
3: RECRUITER SUGGESTIONS
FOR DECREASING CHEATING
AND REPEATING
56
Screener Questions
•
Cheaters who lie to get into qualitative research studies and repeaters who
participate too often are a major concern for the industry as a whole.
•
Closed-ended questions make it easier for respondents to guess how they can
qualify for a study. Some say “no” to all occupations listed in the security question
and “no” to past 6-month participation in order to be recruited.
– Questions in which answers are pre-coded but not read aloud are more difficult for
these respondents to figure out.
“Some of our respondents are experts on qualifying themselves because if you give them
a range, they’ll pick the biggest range.”
“They tend to cheat on income because income is usually a scale.”
“Especially when they’re from the database, then they know [to say “no” to each
occupation].”
“A lot of them will fudge on their answers. A lot of them know it’s 6 months
[participation] and they say, ‘I don’t really know’ and they’ll get it wrong.”
57
Screener Questions
•
Terminating respondents right after a terminate answer enables some to guess
why they do not qualify for the study. This can be sensitive in the case of
demographic quotas.
“I don’t tell them that they’re not qualified. I say ‘that quota’s filled and if it changes
then we’ll call you back’ so they’re not thinking about why they didn’t qualify.”
58
Preventing Cheaters/Repeaters
•
Facility monitoring of recruiting and confirmation calls help to identify respondents
falsifying information and ensures that recruiters follow procedures.
– A facility staff member other than the original recruiter should make the confirmation
calls to double check recruiting.
“It is funny when they ask, ‘What am I supposed to say’ or ‘what did I say the last time?’
And we say, ‘Well, just answer the truth.’ That is a clue to block them in our system.”
59
Preventing Cheaters/Repeaters
•
Keeping the facility database updated is crucial.
– A list of cheaters/repeaters (“professional respondents”) and those who are suspect
(e.g., arrive late on repeated occasions) is used by recruiters.
“Keeping the database updated and being on the lookout for names you recognize. You know these
names are going to pop in your head after a couple of calls.”
“A person will call in with a different name and the same number. We’ll put notes on that person’s profile
and the database manager will delete them from the database.”
“We make a list of people who are professional respondents and keep the list on our desk.”
“ We share information with other facilities when we find a ‘nest of phony respondents.’”
“We keep them in our database and flag them so that they can’t sign up again. We don’t call them. We
don’t recruit people who call us on the phone asking about studies, Everyone has to be checked in our
database.”
“There are also problems with people who are scamming the system, coming only at the time when they
will be paid and sent.”
60
4: RECRUITER SUGGESTIONS
FOR INCREASING
RESPONDENT COOPERATION
61
Recruiting Techniques
•
Good recruiters play a major role in encouraging cooperation:
– Friendly/pleasant yet professional manner.
– Convey a sense of credibility.
– Develop rapport quickly, engage respondents in conversation, but avoid talking too
much, which can sound “sales-y.”
– Upbeat tone of voice, communicating that the study will be interesting/enjoyable.
– Listen to respondents, make them feel special/valued.
– Demeanor appropriate to the study and respondent (more formal with businesspeople).
– Display confidence without coming across too strong.
62
Recruiting Techniques
•
Recruiter manner is professional and personable.
“You have to know what you’re talking about because you have to give them
confidence. They have to feel that this is something that would interest them, that
you’re calling them because they’re important and they’re one of the few people that
might be able to contribute to your study, you have to make them feel special.”
“Be confident. You’ve got to sound like you know what you’re talking about, speak
clearly, and try to be professional.”
“You have to dig in and sell yourself and you have to put that smile in your voice in order
to get through.”
•
Good recruiters follow interviewing protocols:
– Read screener questions verbatim.
– Never lead the respondent.
63
Study Requirements
•
Recruiters stressed that the QRC/client needs to fully explain the screener and
study goals.
– A written explanation of the type of respondent the client is seeking is very helpful – the
screener questions and specifications may not be sufficient.
“I think the client has a part to play in us being informed about the screener. Sometimes
we get screeners and they’re a little obscure, they’re not clear.”
“All project managers should understand their screeners, but they don’t.”
“Sometimes even our client contact does not understand the questions.”
“I have worked with one project manager who really reads the screener and writes
notes. If [something is unclear], he’ll go over it with you.”
“I think that we are sometimes the best people to find mistakes in screeners. Maybe the
skip pattern is wrong or terminates are not marked. We need to understand in broad
strokes who they are looking for so we can see the logic in the screener.”
64
Email Blasts
•
Email blasts to the facility database and/or a client-provided list can be very
effective in pre-screening respondents.
–
–
–
–
Short online surveys, such as Survey Monkey, can be used.
Respondents who pass the initial stage are then formally screened.
This approach is especially useful on low incidence studies and with younger people.
The emails should be worded carefully to avoid biasing respondents.
“With the e-blast you know that this person may potentially qualify versus just pulling
from demographics in the database.”
“It’s easier because you get more people calling in than calling out.”
“With young people, it’s better to email them than call them.”
65
Screener Introductions
•
Introductions are very important in encouraging respondent participate, especially
on contacts with people not in the facility database.
– Explain the purpose of the call and the study clearly/concisely, being as specific as
possible about who, what, when, where and why, etc.
– The study descriptions should sound interesting – the project manager can give advice
on wording.
“I think the opening is the most important part. You must tell them when the group is,
what the subject is, and that you’re not trying to sell them anything.”
“I have to understand what they’re trying to get and then shorten it to let the
respondent know that it’s something interesting to them.”
66
Screener Wording/Length
•
Keep the screener short (under 10 minutes), place terminate questions early, make
skip patterns clear.
–
The screener should be only/mainly questions needed for recruiting, not extraneous
information. If needed, a brief questionnaire can be filled out in the waiting room.
•
Recognize that memories of recent experiences are better than ones a while ago.
•
For children’s research, use simple questions with age-appropriate wording.
– “Do you like SpongeBob a lot or a little?” vs. “on a scale of 1 to 5 where 1 means [etc.]”
– Ask the child first. Parents do not always know what their child watches on TV or what
characters they like. Do not allow parents to prompt the child.
67
Screener Questions
•
Articulation questions should be engaging and appropriate to the study, not
threatening or annoying.
– Respondents can be put off or intimidated by questions requiring knowledge on a
subject irrelevant to the research.
– Certain “creative” questions are considered “dumb” by some respondents (e.g., “name
10 things you can do with a paperclip”).
•
Ethnicity questions and quotas need to keep up with the times.
– It is harder to find “Caucasians” now because so many people are a mix.
– People with Hispanic names may be different from people living the Hispanic culture.
“The best example in the world is Tiger Woods. What ethnicity do you call him? Better
yet, what ethnicity do you call his children?”
“The client asked for no more than 2 ‘Others’ for ethnicity. Everyone is an ‘Other’ these
days.
“I wish they would tell us what they are looking for. I am Hispanic because my great
grandfather on my mother’s side came from South America. I am not Hispanic in the
same sense as some screeners ask for.”
68
Screener Algorithms
•
Algorithms should be avoided if possible. If they must be used, explain the system
to respondents at the start or ask them online in a pre-screener, and keep the rest
of the screener short.
“I tell them upfront that their answers will go through a computer-generated program
after they answer the questions so they are not surprised when I say I need to call them
back. It is a pain though.”
69
Homework
•
Keep homework to a minimum, especially for businesspeople/professionals.
•
Provide appropriate incentives for the pre-work.
– If it is necessary to cancel a respondent who completed homework, pay partial
incentive.
•
If daily Web-based homework is required, provide a telephone number and/or
email address for the facility and respondents if problems arise on a weekend.
“I had people just drop out after a few days because the website did not work. They
were good people but a bad site.”
70
Other Suggestions
•
Location:
– Choose a facility or other place with reasonable time for desired respondents.
•
Interview time
– Schedule interviews for respondents’ convenience.
– Consider weekends and holidays for longer interviews and certain segments.
•
Specifications/quotas
– If possible, use broader specifications so recruiting is not so restrictive.
71
Other Suggestions
•
Incentives
– Compensate respondents fairly for their time and effort.
– A small gift in addition to cash incentives is appreciated; these should be given out at the
end of the interview.
•
Confirmations
– Let respondents on hold know as soon as possible if they are confirmed.
72
APPENDIX:
MODERATOR’S GUIDE
73
Moderator’s Topic Guide
•
•
•
Group 1 - Professional B2B
Group 2 - Consumer Recruiters
Introduction
– Purpose of group
– Who I am and what I do
– What you must do – speak to me
•
Ground Rules
– Audio taping, speak up and loud, one way mirror, observers
•
Respondent Introductions
–
–
–
–
Your first name
Length of time working as a recruiter/working at (company) as a recruiter
Primarily recruit consumer/business
Purpose: determine how, if at all, respondent cooperation rates for qualitative research
have changed and how they could be improved
74
Moderator’s Topic Guide
•
Trends in Respondent Cooperation
– Do respondents seem more or less willing to participate in qualitative research now or a year
ago, or is there no change?
– If a change, what do you feel are the reasons for change?
– Has the economy had an impact on current respondent cooperation rates?
•
Identify Factors that Affect Respondent Cooperation
– What factors make it easier or more difficult to gain respondent cooperation?
– Identify factors that most impact cooperation rates for consumers/business professionals and
how.
•
Type of recruit impact on cooperation rates
– Database, list, or random digit dial, blind recruit versus client identified, etc.
– Any impact of telemarketing, fundraising, “do not call” on respondent cooperation.
•
Respondent type needed impact
 How cooperation rates vary based on age, race, economic status, occupation, experience level, etc.
•
Interview type impact
 Focus group, IDI, shop-a-longs, ethnographies, etc.
75
Moderator’s Topic Guide
– Screener impact
 Length of screener, type of introduction, type of invitation, type and order of questions asked,
use of algorithms, etc.
 Specific screener features that you believe increase or decrease cooperation.
– Time impact
 Length of the interview (2-hour group, 1-hour IDI, several-hour shop-a-long or ethnography,
etc.)
 Time of day/evening the interview is scheduled, etc.
– Topic impact
– Homework projects impact
 By type of homework
– Incentives/honoraria
 Gift, check, cash or value/amount of money, etc.
76
Moderator’s Topic Guide
– Recruiter style impact
 Do certain telephone personality styles increase or decrease cooperation rates?
 Role of recruiter experience.
– Initial email contact impact
– Respondents’ experience in previous research impact
 Do respondents differentiate between survey research and qualitative; if so, how?
 What, if anything, respondents tell recruiters about their experiences.
 How not qualifying for a previous qualitative research study impacts respondent cooperation rates.
•
Obtain Recruiters’ Suggestions for Decreasing the Incidence of Cheating/Repeating
– Do you ever suspect that a respondent is not answering the questions on a screener honestly;
if so, how?
•
Obtain Recruiters’ Suggestions for Increasing Respondent Cooperation Rates
– For list/random dial recruits, for consumer recruits, for business executive/professional
recruits.
– Suggestions for screener design, recruiter training programs, etc.
– Prioritize the one or two most important things qualitative researchers could do to improve
rates.
77